Optimization

Bunch of optimizer implementations in PyTorch

Bunch of optimizer implementations in PyTorch with clean-code, strict types. Also, including useful optimization ideas. Most of the implementations are based on the original paper, but I added some tweaks. Documentation https://pytorch-optimizers.readthedocs.io/en/latest/ Usage Install $ pip3 install pytorch-optimizer Simple Usage from pytorch_optimizer import Ranger21 … model = YourModel() optimizer = Ranger21(model.parameters()) … for input, output in data: optimizer.zero_grad() loss = loss_function(output, model(input)) loss.backward() optimizer.step() Supported Optimizers Useful Resources Several optimization ideas to regularize & stabilize the training. Most of the […]

Read more
Deep Learning

A Simple Baseline for Bayesian Uncertainty in Deep Learning

TensorFlow implementation of “A Simple Baseline for Bayesian Uncertainty in Deep Learning” Concept Algorithm to utilize the SWAG [1]. Equation for the weight sampling from SWAG [1]. Results The red color and the blue color represent the initial state and current state respectively. Performance MNIST Method Accuracy Precision Recall F1-Score Final Epoch 0.99230 0.99231 0.99222 0.99226 Best Loss 0.99350 0.99350 0.99338 0.99344 SWAG (S = 30) 0.99310 0.99305

Read more
GPU

High performance ptychography reconstruction python package running on GPU

Quickstart Need to install python3 to run the GUI and ptychopy, other needed library isin requirement.txt.(Tested OS RHEL 6.0, 7.0). This library could also becompiled as a CUDA-C library. Inside src folder, change build.sh with yourHDF library path. Recommend conda virtual environment, for example conda create -n py36 python=3.6 hdf5-external-filter-plugins-lz4 Activate the virtual environment source activate py36 To install and build the python package, set environment variables HDF5_BASEand CUDAHOME, which point to the installed path of the HDF5 and CUDAlibraries. […]

Read more
PyTorch

Compact Bilinear Pooling for PyTorch

This repository has a pure Python implementation of Compact Bilinear Pooling and Count Sketch for PyTorch. This version relies on the FFT implementation provided with PyTorch 0.4.0 onward. For older versions of PyTorch, use the tag v0.3.0. Installation Run the setup.py, for instance: Usage class compact_bilinear_pooling.CompactBilinearPooling(input1_size, input2_size, output_size, h1 = None, s1 = None, h2 = None, s2 = None) Basic usage: from compact_bilinear_pooling import CountSketch, CompactBilinearPooling input_size = 2048 output_size = 16000 mcb = CompactBilinearPooling(input_size, input_size, output_size).cuda() x = […]

Read more
Attention

A Pytorch Implementation of the Transformer: Attention Is All You Need

Our implementation is largely based on Tensorflow implementation Requirements Why This Project? I’m a freshman of pytorch. So I tried to implement some projects by pytorch. Recently, I read the paper Attention is all you need and impressed by the idea. So that’s it. I got similar result compared with the original tensorflow implementation. Differences with the original paper I don’t intend to replicate the paper exactly. Rather, I aim to implement the main ideas in the paper and verify […]

Read more
Sequences

Sequence modeling benchmarks and temporal convolutional networks

This repository contains the experiments done in the work An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling by Shaojie Bai, J. Zico Kolter and Vladlen Koltun. We specifically target a comprehensive set of tasks that have been repeatedly used to compare the effectiveness of different recurrent networks, and evaluate a simple, generic but powerful (purely) convolutional network on the recurrent nets’ home turf. Experiments are done in PyTorch. If you find this repository helpful, please cite […]

Read more
Deep Learning

A Structured Self-attentive Sentence Embedding

Implementation for the paper A Structured Self-Attentive Sentence Embedding, which was published in ICLR 2017: https://arxiv.org/abs/1703.03130 . USAGE: For binary sentiment classification on imdb dataset run : python classification.py “binary” For multiclass classification on reuters dataset run : python classification.py “multiclass” You can change the model parameters in the model_params.json file Other tranining parameters like number of attention hops etc can be configured in the config.json file. If you want to use pretrained glove embeddings , set the use_embeddings parameter […]

Read more
Attention

Pervasive Attention: 2D Convolutional Networks for Sequence-to-Sequence Prediction

This is a fork of Fairseq(-py) with implementations of the following models: Pervasive Attention – 2D Convolutional Neural Networks for Sequence-to-Sequence Prediction An NMT models with two-dimensional convolutions to jointly encode the source and the target sequences. Pervasive Attention also provides an extensive decoding grid that we leverage to efficiently train wait-k models. See README. Efficient Wait-k Models for Simultaneous Machine Translation Transformer Wait-k models (Ma et al., 2019) with unidirectional encoders and with joint training of multiple wait-k paths. […]

Read more
1 … 451 452 453 454 455 … 928