Cat's Eye

Neural network library and applications in C.

日本語

Location	Contents
catseye.h	Shared neural-network library
catseye_llm.h	Shared GGUF/LLM inference runtime
transcript/	Speech transcription CLI, speech/ONNX headers, model tools, Colab notebooks, fixtures and tests
plamo/	PLaMo 3 Translate translation CLI (0.8B, Q4/Q8 quantization)
llm/	Original LLM examples and GPU implementation
example/, super_resolution/, word2vec/	Neural-network examples
tools/	General library tools

Run make help for project commands. make test runs the library tests (test/: finite-difference gradient check of every layer type, batch-norm / augmentation / safetensors / quantization / GEMM checks, small end-to-end training runs). Build transcription with make -C transcript and run it from that directory so its default models/ path resolves there. Use make legacy for the original root training examples.

Features

• Lightweight and minimalistic:
  • Header only
  • Just include catseye.h and write your model in c. There is nothing to install.
  • Small dependency & simple implementation
• Fast: [under construction]
  • OpenCL support (GPGPU)
  • OpenGL support (GPGPU)
  • SSE, AVX support (But gcc and clang support SIMD...)
  • OpenMP support
  • Support half precision floats (16bit)
• Supported networks:
  • Activation functions
    • sigmoid
    • softmax
    • tanh, scaled tanh (1.7519 * tanh(2/3x))
    • ReLU, Leaky ReLU, ELU, RReLU
    • abs
    • identity
  • Loss functions
    • cross-entropy, mean-squared-error
  • Optimization algorithms
    • SGD (stochastic gradient descent) with/without L2 normalization
    • Momentum SGD
    • AdaGrad
    • RMSProp
    • Adam
  • Layer types
    • linear (mlp)
    • convolution
    • convolution 1d
    • deconvolution
    • Sub-Pixel Convolution (Pixel Shuffler)
    • max pooling
    • average pooling
    • global average pooling (GAP)
    • batch normalization (per-channel statistics over the batch; running mean/variance, gamma and beta are stored in the weights and used by CatsEye_forward() / CatsEye_predict())
    • concat
    • shortcut
• Loader formats:
  • PNG
  • cifar [https://www.cs.toronto.edu/~kriz/cifar.html]
  • MNIST
  • Data Augmentation (noise, zoom, translation)

Learning rate and batch size

Every layer (dense, convolution, deconvolution, batch normalization) accumulates the mean gradient over the batch, and the bias is updated by the optimizer together with the weights. The learning rate eta therefore means the same thing in every layer. (Earlier versions averaged only in convolution layers; to keep an old setup, multiply eta of its dense/deconvolution layers by the batch size.)

Usage

Just include header files in your project.

for more information, see example/

$ dnf install ghostscript ocl-icd-devel
$ cd example
$ make
$ ./sin

Demo

• Recognizing handwritten digits by MNIST training (example/mnist_train.c)

  • http://yui0.github.io/catseye/example/html/mnist.html

• Recognizing pictures (example/cifar10_train.c)

  • http://yui0.github.io/catseye/example/html/cifar10.html

• Neural Network 'paints' an image (example/paint.c)

  

  

  

• Function approximation (example/sin.c)

  

• Convolution Autoencoder (example/mnist_cnn_autoencoder.c,example/cifar_autoencoder.c)

  

• DCGAN (example/mnist_lsgan.c)

  • epoch 1900

  

• Autoencoder (example/mnist_autoencoder.c,example/mnist_autoencoder2.c)

  

• VanillaGAN

  

• Convolutional Neural Networks (example/mnist_cnn_train.c)

  • tanh, 7x7, 32ch, 99.2%

  

Question

• Neural Network Always Produces Same/Similar Outputs for Any Input
  • Scale down the problem to manageable size.
  • Make sure you have enough hidden units.
  • Change the activation function and its parameters.
  • Change learning algorithm parameters.

Refrences

• Documents

  • Neural Networks and Deep Learning [http://nnadl-ja.github.io/nnadl_site_ja/chap1.html]

  • Explain easy backpropagation in the universe [https://www.yukisako.xyz/entry/backpropagation]

  • Optimization algorithm with super easy explanation [https://qiita.com/omiita/items/1735c1d048fe5f611f80]

  • Basic parts of calculation graph used for backpropagation method, etc. [https://qiita.com/t-tkd3a/items/031c0a4dbf25fd2866a3]

  • Automatic differentiation [https://tech-lab.sios.jp/archives/21072]

  • Machine learning [http://hokuts.com/category/%E3%83%97%E3%83%AD%E3%82%B0%E3%83%A9%E3%83%A0/%E6%A9%9F%E6%A2%B0%E5%AD%A6%E7%BF%92/]

  • tiny-cnn [https://github.com/nyanp/tiny-cnn/wiki/%E5%AE%9F%E8%A3%85%E3%83%8E%E3%83%BC%E3%83%88]

  • CS231n Convolutional Neural Networks for Visual Recognition [http://cs231n.github.io/neural-networks-3/#anneal]

  • SVM [http://d.hatena.ne.jp/echizen_tm/20110627/1309188711]

  • Autoencoder

    • Summary of research on VAE [https://www.hiro877.com/entry/vae-research]
    • Hello Autoencoder [https://kiyukuta.github.io/2013/08/20/hello_autoencoder.html]
    • Autoencoder [https://pc.atsuhiro-me.net/entry/2015/08/18/003402]
    • Autoencoder [https://www.slideshare.net/at_grandpa/chapter5-50042838]

  • Convolutional Neural Networks [http://blog.yusugomori.com/post/129688163130/%E6%95%B0%E5%BC%8F%E3%81%A7%E6%9B%B8%E3%81%8D%E4%B8%8B%E3%81%99-convolutional-neural-networks-cnn]

  • Backpropagation [http://postd.cc/2015-08-backprop/]

  • Perceptron [http://tkengo.github.io/blog/2015/08/21/visual-perceptron/]

• Programing
  • Multilayer perceptron [http://kivantium.hateblo.jp/entry/2014/12/22/004640]
  • Weather example [http://arakilab.media.eng.hokudai.ac.jp/~t_ogawa/wiki/index.php?LibSVM]
  • Recognizing handwritten digits [http://aidiary.hatenablog.com/entry/20140201/1391218771]
  • Recognizing handwritten digits on Web [http://d.hatena.ne.jp/sugyan/20151124/1448292129]
  • Image generator by Denoising Autoencoder [http://joisino.hatenablog.com/entry/2015/09/09/224157]
  • Neural Network 'paints' an image [http://cs.stanford.edu/people/karpathy/convnetjs/demo/image_regression.html]
• Data
  • UTKFace - Large Scale Face Dataset
  • Caltech 101