Gradient-Based Learning Applied to Document Recognition

Yann LeCun, Leon Bottou, Yoshua Bengio, Patrick Haffner, 1998, Proceedings of the IEEE

About this paperEnglishbeside English

This paper is under a licence that does not allow it to be reproduced, so what is here is its front matter and a short abstract. Read it at the publisher's page.

Mã định danh HAL: hal-03926082

https://hal.science/hal-03926082v1

Nộp ngày 6 tháng 1 năm 2023

HAL là kho lưu trữ truy cập mở đa ngành dành cho việc lưu trữ và phổ biến các tài liệu nghiên cứu khoa học, dù đã được công bố hay chưa. Các tài liệu có thể đến từ các cơ sở giảng dạy và nghiên cứu ở Pháp hoặc nước ngoài, hoặc từ các trung tâm nghiên cứu công lập hay tư nhân.

Kho lưu trữ mở đa ngành HAL dành cho việc lưu trữ và phổ biến các tài liệu khoa học ở cấp độ nghiên cứu, đã hoặc chưa được công bố, đến từ các cơ sở giảng dạy và nghiên cứu của Pháp hoặc nước ngoài, các phòng thí nghiệm công lập hay tư nhân.

Sự cho phép của HAL

YANN LECUN, THÀNH VIÊN IEEE, LÉON BOTTOU, YOSHUA BENGIO, VÀ PATRICK HAFFNER

Mạng nơ-ron (neural network) nhiều lớp được huấn luyện bằng thuật toán lan truyền ngược (back-propagation) là ví dụ tiêu biểu nhất về một kỹ thuật học dựa trên gradient thành công. Với một kiến trúc mạng phù hợp, các thuật toán học dựa trên gradient có thể được dùng để tạo ra một mặt quyết định phức tạp có khả năng phân loại các mẫu nhiều chiều, chẳng hạn như các ký tự viết tay, với mức tiền xử lý tối thiểu. Bài báo này tổng quan các phương pháp khác nhau được áp dụng cho nhận dạng ký tự viết tay và so sánh chúng trên một tác vụ nhận dạng chữ số viết tay tiêu chuẩn. Mạng nơ-ron tích chập (convolutional neural network), được thiết kế riêng để xử lý sự biến thiên của các hình dạng hai chiều (2-D), được chứng minh là vượt trội hơn tất cả các kỹ thuật khác.

*Các hệ thống nhận dạng tài liệu trong thực tế gồm nhiều mô-đun, bao gồm trích xuất trường, phân đoạn, nhận dạng và mô hình hóa ngôn ngữ.

References

  1. R. O. Duda and P. E. Hart, Pattern Classification and Scene Analysis. New York: Wiley, 1973.
  2. Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, “Backpropagation applied to handwritten zip code recognition,” Neural Computation, vol. 1, no. 4, pp. 541–551, Winter 1989.
  3. S. Seung, H. Sompolinsky, and N. Tishby, “Statistical mechanics of learning from examples,” Phys. Rev. A, vol. 45, pp. 6056–6091, 1992.
  4. V. N. Vapnik, E. Levin, and Y. LeCun, “Measuring the vc-dimension of a learning machine,” Neural Computation, vol. 6, no. 5, pp. 851–876, 1994.
  5. C. Cortes, L. Jackel, S. Solla, V. N. Vapnik, and J. Denker, “Learning curves: Asymptotic values and rate of convergence,” in Advances in Neural Information Processing Systems 6, J. D. Cowan, G. Tesauro, and J. Alspector, Eds. San Mateo, CA: Morgan Kaufmann, 1994, pp. 327–334.
  6. V. N. Vapnik, The Nature of Statistical Learning Theory. New York: Springer, 1995.
  7. ____, Statistical Learning Theory. New York: Wiley, 1998.
  8. W. H. Press, B. P. Flannery, S. A. Teukolsky, and W. T. Vetterling, Numerical Recipes: The Art of Scientific Computing. Cambridge, UK: Cambridge Univ., 1986.
  9. S. I. Amari, “A theory of adaptive pattern classifiers,” IEEE Trans. Electron. Comput., vol. EC-16, pp. 299–307, 1967.
  10. Y. Tsypkin, Adaptation and Learning in Automatic Systems New York: Academic, 1971.
  11. ____, Foundations of the Theory of Learning Systems. New York: Academic, 1973.
  12. M. Minsky and O. Selfridge, “Learning in random nets,” in Proc. 4th London Symp. Information Theory, pp. 335–347, 1961.
  13. D. H. Ackley, G. E. Hinton, and T. J. Sejnowski, “A learning algorithm for Boltzmann machines,” Cognitive Sci., vol. 9, pp. 147–169, 1985.
  14. G. E. Hinton and T. J. Sejnowski, “Learning and relearning in Boltzmann machines,” in Parallel Distributed Processing: Explorations in the Microstructure of Cognition. Volume 1: Foundations, D. E. Rumelhart and J. L. McClelland, Eds. Cambridge, MA: MIT, 1986.
  15. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning internal representations by error propagation,” in Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol. I. Cambridge, MA: Bradford Books, 1986, pp. 318–362,
  16. A. E. Bryson, Jr. and Y.-C. Ho, Applied Optimal Control. London, UK: Blaisdell, 1969.
  17. Y. LeCun, “A learning scheme for asymmetric threshold networks,” in Proc. Cognitiva ’85, Paris, France, 1985, pp. 599–604.
  18. ____, “Learning processes in an asymmetric threshold network,” in Disordered Systems and Biological Organization, E. Bienenstock, F. Fogelman-Soulië, and G. Weisbuch, Eds. Les Houches, France: Springer-Verlag, 1986, pp. 233–240.
  19. D. B. Parker, “Learning-logic,” Sloan School Manage., MIT, Cambridge, MA, Tech. Rep., TR-47, Apr. 1985.
  20. Y. LeCun, Modèles Connexionnistes de l’Apprentissage (Connectionist Learning Models), Ph.D. dissertation, Université P. et M. Curie (Paris 6), June 1987.
  21. ____, “A theoretical framework for back-propagation,” in Proc. 1988 Connectionist Models Summer School, D. Touretzky, G. Hinton, and T. Sejnowski, Eds. Pittsburgh, PA: CMU, Morgan Kaufmann, 1988, pp. 21–28.
  22. L. Bottou and P. Gallinari, “A framework for the cooperation of learning algorithms,” in Advances in Neural Information Processing Systems, vol. 3, D. Touretzky and R. Lippmann, Eds. Denver, CO: Morgan Kaufmann, 1991.
  23. C. Y. Suen, C. Nadal, R. Legault, T. A. Mai, and L. Lam, “Computer recognition of unconstrained handwritten numerals,” Proc. IEEE, vol. 80, pp. 1162–1180, July 1992.
  24. S. N. Srihari, “High-performance reading machines,” Proc. IEEE., vol. 80, pp. 1120–1132, July 1992.
  25. Y. LeCun, L. D. Jackel, B. Boser, J. S. Denker, H. P. Graf, I. Guyon, D. Henderson, R. E. Howard, and W. Hubbard, “Handwritten digit recognition: Applications of neural net chips and automatic learning,” IEEE Trans. Commun., vol. 37, pp. 41–46, Nov. 1989.
  26. J. Keeler, D. Rumelhart, and W. K. Leow, “Integrated segmentation and recognition of hand-printed numerals,” in Neural Information Processing Systems, R. P. Lippmann, J. M. Moody, and D. S. Touretzky, Eds. San Mateo, CA: Morgan Kaufmann, vol. 3, pp. 557–563, 1991.
  27. O. Matan, C. J. C. Burges, Y. LeCun, and J. S. Denker, “Multi-digit recognition using a space displacement neural network,” vol. 4, in Neural Information Processing Systems, J. M. Moody, S. J. Hanson, and R. P. Lippman, Eds. San Mateo, CA: Morgan Kaufmann, 1992.
  28. L. R. Rabiner, “A tutorial on hidden Markov models and selected applications in speech recognition,” Proc. IEEE, vol. 77, pp. 257–286, Feb. 1989.
  29. H. A. Bourland and N. Morgan, Connectionist Speech Recognition: A Hybrid Approach. Boston: Kluwer, 1994.
  30. D. H. Hubel and T. N. Wiesel, “Receptive fields, binocular interaction, and functional architecture in the cat’s visual cortex,” J. Physiology (London), vol. 160, pp. 106–154, 1962.
  31. K. Fukushima, “Cognition: A self-organizing multilayered neural network,” Biological Cybern., vol. 20, pp. 121–136, 1975.
  32. K. Fukushima and S. Miyake, “Neocognitron: A new algorithm for pattern recognition tolerant of deformations and shifts in position,” Pattern Recognit., vol. 15, no. 6, pp. 455–469, Nov. 1982.
  33. M. C. Mozer, The Perception of Multiple Objects: A Connectionist Approach. Cambridge, MA: MIT-Bradford Books, 1991.
  34. Y. LeCun, “Generalization and network design strategies,” in Connectionism in Perspective, R. Pfeifer, Z. Schreter, F. Fogelman, and L. Steels, Eds. Zurich, Switzerland: Elsevier, 1989.
  35. Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, “Handwritten digit recognition with a back-propagation network,” in Advances in Neural Information Processing Systems 2 (NIPS’89), David Touretzky, Ed. Denver, CO: Morgan Kaufmann, 1990.
  36. G. L. Martin, “Centered-object integrated segmentation and recognition of overlapping hand-printed characters,” Neural Computation, vol. 5, no. 3, pp. 419–429, 1993.
  37. J. Wang and J. Jean, “Multi-resolution neural networks for omnifont character recognition,” in Proc. Int. Conf. Neural Networks, vol. III, 1993, pp. 1588–1593.
  38. Y. Bengio, Y. LeCun, C. Nohl, and C. Burges, “Lerec: A NN/HMM hybrid for on-line handwriting recognition,” Neural Computation, vol. 7, no. 5, 1995.
  39. S. Lawrence, C. L. Giles, A. C. Tsoi, and A. D. Back, “Face recognition: A convolutional neural network approach,” IEEE Trans. Neural Networks, vol. 8, pp. 98–113, Jan. 1997.
  40. K. J. Lang and G. E. Hinton, “A time delay neural network architecture for speech recognition,” Carnegie-Mellon Univ., Pittsburgh, PA, Tech. Rep. CMU-CS-88-152, 1988.
  41. A. H. Waibel, T. Hanazawa, G. Hinton, K. Shikano, and K. Lang, “Phoneme recognition using time-delay neural networks,” IEEE Trans. Acoustics, Speech, Signal Processing, vol. 37, pp. 328–339, Mar. 1989.
  42. L. Bottou, F. Fogelman, P. Blanchet, and J. S. Lienard, “Speaker independent isolated digit recognition: Multilayer perceptron versus dynamic time warping,” Neural Networks, vol. 3, pp. 453–465, 1990.
  43. P. Haffner and A. H. Waibel, “Time-delay neural networks embedding time alignment: A performance analysis,” in Proc. EUROSPEECH’91, 2nd Europ. Conf. Speech Communication and Technology, Genova, Italy.
  44. I. Guyon, P. Albrecht, Y. LeCun, J. S. Denker, and W. Hubbard, “Design of a neural network character recognizer for a touch terminal,” Pattern Recognit., vol. 24, no. 2, pp. 105–119, 1991.
  45. J. Bromley, J. W. Bentz, L. bottou, I. Guyon, Y. LeCun, C. Moore, E. Säckinger, and R. Shah, “Signature verification using a siamese time delay neural network,” Int. J. Pattern Recognit. Artificial Intell., vol. 7, no. 4, pp. 669–687, Aug. 1993.
  46. Y. LeCun, I. Kanter, and S. Solla, “Eigenvalues of covariance matrices: Application to neural-network learning,” Phys. Rev. Lett., vol. 66, no. 18, pp. 2396–2399, May 1991.
  47. T. G. Dietterich and G. Bakiri, “Solving multiclass learning problems via error-correcting output codes,” J. Artificial Intell. Res., vol. 2, pp. 263–286, 1995.
  48. L. R. Bahl, P. F. Brown, P. V. de Souza, and R. L. Mercer, “Maximum mutual information of hidden Markov model parameters for speech recognition,” in Proc. Int. Conf. Acoustics, Speech, Signal Processing, 1986, pp. 49–52.
  49. ____, “Speech recognition with continuous-parameter hidden Markov models,” Comput., Speech Language, vol. 2, pp. 219–234, 1987.
  50. B. H. Juang and S. Katagiri, “Discriminative learning for minimum error classification,” IEEE Trans. Acoustics, Speech, Signal Processing, vol. 40, pp. 3043–3054, Dec. 1992.
  51. Y. LeCun, L. D. Jackel, L. Bottou, A. Brunot, C. Cortes, J. S. Denker, H. Drucker, I. Guyon, U. A. Muller, E. Säckinger, P. Simard, and V. N. Vapnik, “Comparison of learning algorithms for handwritten digit recognition,” in Int. Conf. Artificial Neural Networks, F. Fogelman and P. Gallinari, Eds. Paris: EC2 & Cie, 1995, pp. 53–60.
  52. I. Guyon, I. Poujaud, L. Personnaz, G. Dreyfus, J. Denker, and Y. LeCun, “Comparing different neural net architectures for classifying handwritten digits,” in Proc. IEEE IJCNN, Washington, DC, vol. II, 1989, pp. 127–132..
  53. R. Ott, “Construction of quadratic polynomial classifiers,” in Proc. IEEE Int. Conf. Pattern Recognition, 1976, pp. 161–165.
  54. J. Schürmann, “A multifont word recognition system for postal address reading,” IEEE Trans. Comput., vol. C-27, pp. 721–732, Aug. 1978.
  55. Y. Lee, “Handwritten digit recognition using k-nearest neighbor, radial-basis functions, and backpropagation neural networks,” Neural Computation, vol. 3, no. 3, pp. 440–449, 1991.
  56. D. Saad and S. A. Solla, “Dynamics of on-line gradient descent learning for multilayer neural networks,” in Advances in Neural Information Processing Systems, vol. 8, D. S. Touretzky, M. C. Mozer, and M. E. Hasselmo, Eds. Cambridge, MA: MIT, 1996, pp. 302–308.
  57. G. Cybenko, “Approximation by superpositions of sigmoidal functions,” Math. Control, Signals, Syst., vol. 2, no. 4, pp. 303–314, 1989.
  58. L. Bottou and V. N. Vapnik, “Local learning algorithms,” Neural Computation, vol. 4, no. 6, pp. 888–900, 1992.
  59. R. E. Schapire, “The strength of weak learnability,” Machine Learning, vol. 5, no. 2, pp. 197–227, 1990.
  60. H. Drucker, R. Schapire, and P. Simard, “Improving performance in neural networks using a boosting algorithm,” in Advances in Neural Information Processing Systems 5, S. J. Hanson, J. D. Cowan, and C. L. Giles, Eds. San Mateo, CA: Morgan Kaufmann, 1993, pp. 42–49.
  61. P. Simard, Y. LeCun, and J. Denker, “Efficient pattern recognition using a new transformation distance,” in Advances in Neural Information Processing Systems, vol. 5, S. Hanson, J. Cowan, and L. Giles, Eds. San Mateo, CA: Morgan Kaufmann, 1993.
  62. B. Boser, I. Guyon, and V. Vapnik, “A training algorithm for optimal margin classifiers,” in Proc. 5th Annu. Workshop Computational Learning Theory, vol. 5, 1992, pp. 144–152.
  63. C. J. C. Burges and B. Schoelkopf, “Improving the accuracy and speed of support vector machines,” in Advances in Neural Information Processing Systems 9, M. Jordan, M. Mozer, and T. Petsche, Eds. Cambridge, MA: MIT, 1997.
  64. E. Säckinger, B. Boser, J. Bromley, Y. LeCun, and L. D. Jackel, “Application of the ANNA neural network chip to high-speed character recognition,” IEEE Trans. Neural Networks, vol. 3, no. 3, pp. 498–505, Mar. 1992.
  65. J. S. Bridle, “Probabilistic interpretation of feedforward classification networks outputs, with relationship to statistical pattern recognition,” in Neurocomputing, Algorithms, Architectures and Applications, F. Fogelman, J. Herault, and Y. Burnod, Eds. Les Arcs, France: Springer, 1989.
  66. Y. LeCun, L. Bottou, and Y. Bengio, “Reading checks with graph transformer networks,” in Proc. IEEE Int. Conf. Acoustics, Speech, Signal Processing. Munich, Germany, vol. 1, 1997, pp. 151–154..
  67. Y. Bengio, Neural Networks for Speech and Sequence Recognition. London, UK: International Thompson, 1996.
  68. C. Burges, O. Matan, Y. LeCun, J. Denker, L. Jackel, C. Stenard, C. Nohl, and J. Ben, “Shortest path segmentation: A method for training a neural network to recognize character strings,” in Proc. Int. Joint Conf. Neural Networks, Baltimore, MD, vol. 3, 1992, pp. 165–172.
  69. T. M. Breuel, “A system for the off-line recognition of handwritten text,” in Proc. IEEE ICPR’94, Jerusalem, pp. 129–134.
  70. A. Viterbi, “Error bounds for convolutional codes and an asymptotically optimum decoding algorithm,” IEEE Trans. Inform. Theory, vol. 15, pp. 260–269, Apr. 1967.
  71. R. P. Lippmann and B. Gold, “Neural-net classifiers useful for speech recognition,” in Proc. IEEE 1st Int. Conf. Neural Networks, San Diego, CA, June 1987, pp. 417–422.
  72. H. Sakoe, R. Isotani, K. Yoshida, K. Iso, and T. Watanabe, “Speaker-independent word recognition using dynamic programming neural networks,” in Proc. Int. Conf. Acoustics, Speech, Signal Processing, Glasgow, 1989, pp. 29–32.
  73. J. S. Bridle, “Alphanets: A recurrent ‘neural’ network architecture with a hidden Markov model interpretation,” Speech Commun., vol. 9, no. 1, pp. 83–92, 1990.
  74. M. A. Franzini, K. F. Lee, and A. H. Waibel, “Connectionist viterbi training: A new hybrid method for continuous speech recognition,” in Proc. Int. Conf. Acoustics, Speech, Signal Processing, Albuquerque, NM, 1990, pp. 425–428.
  75. L. T. Niles and H. F. Silverman, “Combining hidden Markov models and neural network classifiers,” in Proc. Int. Conf. Acoustics, Speech, Signal Processing, Albuquerque, NM, 1990, pp. 417–420.
  76. X. Driancourt and L. Bottou, “MLP, LVQ and DP: Comparison & cooperation,” in Proc. Int. Joint Conf. Neural Networks, Seattle, WA, vol. 2, 1991, pp. 815–819.
  77. Y. Bengio, R. De Mori, G. Flammaia, and R. Kompe, “Global optimization of a neural network-hidden Markov model hybrid,” IEEE Trans. Neural Networks, vol. 3, pp. 252–259, March 1992.
  78. P. Haffner and A. H. Waibel, “Multi-state time-delay neural networks for continuous speech recognition,” vol. 4, in Advances in Neural Information Processing Systems. San Mateo, CA: Morgan Kaufmann, pp. 579–588, 1992.
  79. Y. Bengio, P. Simard, and P. Frasconi, “Learning long-term dependencies with gradient descent is difficult,” IEEE Trans. Neural Networks, vol. 5, no. 2, pp. 157–166, Mar. 1994.
  80. T. Kohonen, G. Barna, and R. Chrisley, “Statistical pattern recognition with neural network: Benchmarking studies,” in Proc. IEEE 2nd Int. Conf. Neural Networks, San Diego, CA, vol. 1, 1988, pp. 61–68.
  81. P. Haffner, “Connectionist speech recognition with a global MMI algorithm,” in Proc. EUROSPEECH’93, 3rd Europ. Conf. Speech Communication and Technology, Berlin, pp. 1929–1932.
  82. J. S. Denker and C. J. Burges, “Image segmentation and recognition,” in The Mathematics of Induction. Reading, MA: Addison Wesley, 1995.
  83. L. Bottou, Une Approche théorique de l’Apprentissage Connexionniste: Applications à la Reconnaissance de la Parole, Ph.D. dissertation, Univ. Paris XI, France, 1991.
  84. M. Rahim, Y. Bengio, and Y. LeCun, “Discriminative feature and model design for automatic speech recognition,” in Proc. Eurospeech, Rhodes, Greece, 1997, pp. 75–78.
  85. U. Bodenhausen, S. Manke, and A. Waibel, “Connectionist architectural learning for high performance character and speech recognition,” in Proc. Int. Conf. Acoustics, Speech, Signal Processing, Minneapolis, MN, vol. 1, 1993, pp. 625–628.
  86. F. Pereira, M. Riley, and R. Sproat, “Weighted rational transductions and their application to human language processing,” in ARPA Natural Language Processing Workshop, 1994.
  87. M. Lades, J. C. Vorbrüggen, J. Buhmann, and C. von der Malsburg, “Distortion invariant object recognition in the dynamic link architecture,” IEEE Trans. Comput., vol. 42, pp. 300–311, March 1993.
  88. B. Boser, E. Säckinger, J. Bromley, Y. LeCun, and L. Jackel, “An analog neural network processor with programmable topology,” IEEE J. Solid-State Circuits, vol. 26, pp. 2017–2025, Dec. 1991.
  89. M. Schenkel, H. Weissman, I. Guyon, C. Nohl, and D. Henderson, “Recognition-based segmentation of on-line hand-printed words,” in Advances in Neural Information Processing Systems 5, S. J. Hanson, J. D. Cowan, and C. L. Giles, Eds. Denver, CO: Morgan Kaufmann, 1993, pp. 723–730.
  90. C. Dugust, L. Devillers, and X. Aubert, “Combining TDNN and HMM in a hybrid system for improved continuous-speech recognition,” IEEE Trans. Speech Audio Processing, vol. 2, pp. 217–224, Jan. 1994.
  91. O. Matan, H. S. Baird, J. Bromley, C. J. C. Burges, J. S. Denker, L. D. Jackel, Y. LeCun, E. P. D. Pednault, W. Satterfield, C. E. Stenard, and T. J. Thompson, “Reading handwritten digits: A ZIP code recognition system,” IEEE Trans. Comput., vol. 25, no. 7, pp. 59–63, July 1992.
  92. Y. Bengio and Y. LeCun, “Word normalization for on-line handwritten word recognition,” in Proc. IEEE Int. Conf. Pattern Recognition, Jerusalem, 1994.
  93. R. Vaillant, C. Monrocq, and Y. LeCun, “Original approach for the localization of objects in images,” Proc. Inst. Elect. Eng., vol. 141, no. 4, pp. 245–250, Aug. 1994.
  94. R. Wolf and J. Platt, “Postal address block location using a convolutional locator network,” in Advances in Neural Information Processing Systems 6, J. D. Cowan, G. Tesauro, and J. Alspector, Eds. San Mateo, CA: Morgan Kaufmann, 1994, pp. 745–752.
  95. S. Nowlan and J. Platt, “A convolutional neural network hand tracker,” in Advances in Neural Information Processing Systems 7, G. Tesauro, D. Touretzky, and T. Leen, Eds. San Mateo, CA: Morgan Kaufmann, 1995, pp. 901–908.
  96. H. A. Rowley, S. Baluja, and T. Kanade, “Neural network-based face detection,” in Proc. IEEE CVPR’96, pp. 203–208.
  97. E. Osuna, R. Freund, and F. Girosi, “Training support vector machines: An application to face detection,” in Proc. IEEE CVPR’96, pp. 130–136.
  98. H. Bourlard and C. J. Wellekens, “Links between Markov models and multilayer perceptrons,” in Advances in Neural Information Processing Systems, D. Touretzky, Ed. Denver: Morgan-Kaufmann, vol. 1, 1989, pp. 186–187.
  99. Y. Bengio, R. De Mori, G. Flammia, and R. Kompe, “Neural network—Gaussian mixture hybrid for speech recognition or density estimation,” in Advances in Neural Information Processing Systems 4, J. E. Moody, S. J. Hanson, and R. P. Lippmann, Eds. Denver, CO: Morgan Kaufmann, 1992, pp. 175–182.
  100. F. C. N. Pereira and M. Riley, “Speech recognition by composition of weighted finite automata,” in Finite-State Devices for Natural Lague Processing. Cambridge, MA: MIT, 1997.
  101. M. Mohri, “Finite-state transducers in language and speech processing,” Computational Linguistics, vol. 23, no. 2, pp. 269–311, 1997.
  102. I. Guyon, M. Schenkel, and J. Denker, “Overview and synthesis of on-line cursive handwriting recognition techniques,” in Handbook on Optical Character Recognition and Document Image Analysis, P. S. P. Wang and H. Bunke, Eds. New York: World Scientific, 1996.
  103. M. Mohri and M. Riley, “Weighted determinization and minimization for large vocabulary recognition,” in Proc. Eurospeech ’97, Rhodes, Greece, pp. 131–134.
  104. Y. Bengio and P. Frasconi, “An input/output HMM architecture,” in Advances in Neural Information Processing Systems, vol. 7, G. Tesauro, D. Touretzky, and T. Leen, Eds. Cambridge, MA: MIT, pp. 427–434, 1996.
  105. ——, “Input/output HMM’s for sequence processing,” IEEE Trans. Neural Networks, vol. 7, no. 5, pp. 1231–1249, 1996.
  106. M. Mohri, F. C. N. Pereira, and M. Riley, A Rational Design for a Weighted Finite-State Transducer Library (Lecture Notes in Computer Science). New York: Springer Verlag, 1997.
  107. M. Rahim, C. H. Lee, and B. H. Juang, “Discriminative utterance verification for connected digits recognition,” IEEE Trans. Speech Audio Processing, vol. 5, pp. 266–277, 1997.
  108. M. Rahim, Y. Bengio, and Y. LeCun, “Discriminative feature and model design for automatic speech recognition,” in Proc. Eurospeech ’97, Rhodes, Greece.
  109. S. Bengio and Y. Bengio, “An EM algorithm for asynchronous input/output hidden Markov models,” in Proc. International Conference on Neural Information Processing, Hong-King, 1996, pp. 328–334.
  110. C. Tappert, C. Suen, and T. Wakahara, “The state of the art in on-line handwriting recognition,” IEEE Trans. Pattern Anal. Machine Intell., vol. 8, pp. 787–808, Dec. 1990.
  111. S. Manke and U. Bodenhausen, “A connectionist recognizer for on-line cursive handwriting recognition,” in Proc. Int. Conf. Acoustics, Speech, Signal Processing, Adelaide, vol. 2, 1994, pp. 633–636.
  112. M. Gilloux and M. Leroux, “Recognition of cursive script amounts on postal checks,” in Proc. Europ. Conf. Postal Technol., Nantes, France, June 1993, pp. 705–712.
  113. D. Guillevic and C. Y. Suen, “Cursive script recognition applied to the processing of bank checks,” in Proc. Int. Conf. Document Analysis Recognition, Montreal, Canada, Aug. 1995, pp. 11–14.
  114. L. Lam, C. Y. Suen, D. Guillevic, N. W. Strathy, M. Cheriet, K. Liu, and J. N. Said, “Automatic processing of information on checks,” in Int. Conf. Systems, Man, and Cybernetics, Vancouver, Canada, Oct. 1995, pp. 2353–2358.
  115. C. J. C. Burges, J. I. Ben, J. S. Denker, Y. LeCun, and C. R. Nohl, “Off line recognition of handwritten postal words using neural networks,” Int. J. Pattern Recognit. Artificial Intell., vol. 7, no. 4, p. 689, 1993.
  116. Y. LeCun, Y. Bengio, D. Henderson, A. Weisbuch, H. Weissman, and L. Jackel, “On-line handwriting recognition with neural networks: Spatial representation versus temporal representation,” in Proc. Int. Conf. Handwriting Drawing, 1993.
  117. U. Müller, A. Gunzinger, and W. Guggenbühl, “Fast neural net simulation with a DSP processor array,” IEEE Trans. Neural Networks, vol. 6, pp. 203–213, Jan. 1995.
  118. R. Battiti, “First- and second-order methods for learning: Between steepest descent and Newton’s method,” Neural Computation, vol. 4, no. 2, pp. 141–166, 1992.
  119. A. H. Kramer and A. Sangiovanni-Vincentelli, “Efficient parallel learning algorithms for neural networks,” in Advances in Neural Information Processing Systems, vol. 1, D. S. Touretzky, Ed. San Mateo, CA: Morgan Kaufmann, 1988, pp. 40–48.
  120. M. Moller, Efficient Training of Feed-Forward Neural Networks, Ph.D. dissertation, Aarhus Univ., Aarhus, Denmark, 1993.
  121. S. Becker and Y. LeCun, “Improving the convergence of back-propagation learning with second-order methods,” Univ. Toronto Connectionist Res. Group, Toronto, Ontario, Canada, Tech. Rep. CRG-TR-88-5, Sept. 1988. Yann LeCun (Member, IEEE) received the Diplôme d’Ingénieur degree from the Ecole Supérieure d’Ingénieur en Electrotechnique et Electronique, Paris, in 1983 and the Ph.D. degree in computer science from the Université Pierre et Marie Curie, Paris, in 1987. During his time at the Université Pierre et Marie Curie, he proposed an early version of the back-propagation learning algorithm for neural networks. He joined the Department of Computer Science at the University of Toronto, Toronto, Ont., Canada, as a Research Associate in 1987. In 1988, he joined the Adaptive Systems Research Department at AT&T Bell Laboratories, Holmdel, NJ, where he worked on neural networks, machine learning, and handwriting recognition. In 1996 he became Head of the Image Processing Services Research Department at AT&T Labs-Research, Red Bank, NJ. He has published over 70 technical papers and book chapters on neural networks, machine learning, pattern recognition, handwriting recognition, document understanding, image processing, very large scale integration (VLSI) design, and information theory. In addition to the above topics, his current interests include video-based user interfaces, image compression, and content-based indexing of multimedia material. Dr. LeCun serves on the board of the Machine Learning Journal and has served as Associate Editor of the IEEE Transactions on Neural Networks. He is General Chair of the “Machines That Learn” workshop, which has been held every year since 1986 in Snowbird, UT. He has served as Program Co-Chair of IJCNN’89, INNC’90, and NIPS’90, ’94, and ’95. He is a member of the IEEE Neural Network for Signal Processing Technical Committee. Léon Bottou received the Dipôme degree from Ecole Polytechnique, Paris, in 1987, the Magistère en Mathématiques Fondamentales et Appliquées et Informatiques degree from Ecole Normale Supérieure, Paris in 1988, and the Ph.D. degree in computer science from Université de Paris-Sud in 1991. During his time at Université de Paris-Sud he worked on speech recognition and proposed a framework for stochastic gradient learning and global training. He then joined the Adaptive Systems Research Department at AT&T Bell Laboratories, Holmdel, NJ, where he worked on neural networks, statistical learning theory, and local learning algorithms. He returned to France in 1992 as a Research Engineer at ONERA. He then became Chairman of Neuristique S.A., a company that makes neural network simulators and traffic forecasting software. He returned to AT&T Bell Laboratories in 1995 where he worked on graph transformer networks for optical character recognition. He is now a Member of the Image Processing Services Research Department at AT&T Labs-Research, Red Bank, NJ. Besides learning algorithms, his current interests include arithmetic coding, image compression, and indexing. Yoshua Bengio received the B.Eng. degree in electrical engineering and the M.Sc. and Ph.D. degrees in computer science from McGill University, Montreal, P.Q., Canada, in 1986, 1988, and 1991, respectively. In 1991–1992 he was a Postdoctoral Fellow at the Massachusetts Institute of Technology, Cambridge. In 1992 he joined AT&T Bell Laboratories, which later became AT&T Labs-Research, Red Bank, NJ. In 1993 he joined the faculty of the computer science department of the Université de Montréal, Montréal, P.Q., Canada, where he is now an Associate Professor. Since his first work on neural networks in 1986, his research interests have been centered around learning algorithms, especially for data with a sequential or spatial nature, such as speech, handwriting, and time-series. Patrick Haffner graduated from Ecole Polytechnique, Paris, in 1987 and from Ecole Nationale Supérieure des Télécommunications (ENST), Paris, in 1989. He received the Ph.D degree in speech and signal processing from ENST in 1994. In 1988 and 1990, he worked on the design of the TDNN and the MS-TDNN architectures at ATR (Japan) and Carnegie Mellon University. From 1989 to 1995, as a Research Scientist for CNET/France-Télécom in Lannion, France, he developed connectionist learning algorithms for telephone speech recognition. In 1995, he joined AT&T Bell Laboratories, Holmdel, NJ, and worked on the application of optical character recognition and transducers to the processing of financial documents. In 1997, he joined the Image Processing Services Research Department at AT&T Labs-Research, Red Bank, NJ. His research interests include statistical and connectionist models for sequence recognition, machine learning, speech and image recognition, and information theory.

Read off the page by a vision model reading the page images using gpt-5. Translated into Tiếng Việt by gpt-6-astra, against glossary version 6. Translated from the English.