Akshay Aggarwal, Saurabh Birari, Venkatesh Lokare, Karthik Rajendran
Introduction
Neural Machine Translation has had a profound impact on our ability to access and understand text in other languages. It has been applied very successfully to tasks that involve translating from one language to another, and we believe that Neural Machine Translation can be used to make our lives even better, even when we don’t need to translate from one language to another.
The goal of this project is to use Neural Machine Translation to translate within a given language. We want to make written communication effortless and frictionless by creating a system that can transform poorly written English sentences to their publication-ready counterparts. This system is exactly what we propose in this blog post - a system that can close the expressivity gap between experienced and inexperienced writers by automatically correcting common (and lesser-known) pitfalls in clear, concise writing.
And, we want to train this model without parallel corpora, i.e. here, we propose an unsupervised mechanism for intra-language machine translation. This project parallels other popular ideas in Deep Learning, notably those of style transfer in images and video, and brings these capabilities to the Natural Language domain. The power of unsupervised learning in this setting is that we can freely acquire large training datasets, and we have the ability to reuse the proposed model for a vast range of like-minded tasks that may not have labeled training data readily available.
While we are still experimenting and training our proposed models, we are optimistic about the profound impact that our work can have in the field of Natural Language Processing and in our abilities to communicate effectively.
In the sections that follow, we will introduce you to the problem with some examples, then we will discuss our proposed architecture and detail the inputs and outputs for our model. After that, we’ll give you a primer on similar work from which we draw inspiration and which validate our ideas. Next, we’ll tell you about the project’s status along with some preliminary results, and, finally, we’ll complete this post with details about our ongoing efforts in the days and weeks to come. Our decided aim is to report our final results by December 23rd before we take time off for the holidays.
Project Goals
The goal of this project is to use Neural Machine Translation to translate within a given language. We want to make written communication effortless and frictionless by creating a system that can transform poorly written English sentences to their publication-ready counterparts. This system is exactly what we propose in this blog post - a system that can close the expressivity gap between experienced and inexperienced writers by automatically correcting common (and lesser-known) pitfalls in clear, concise writing.
To illustrate what we want to achieve, here are some example inputs and outputs of our model.
Input: All of past findings, helps the goal of the study happening now.
Output: The study being conducted benefits from past findings.
Input: Went to Barcelona for the wknd. TTYL about it.
Output: We went to Barcelona for the weekend. I’ll talk to you later about it.
Input: I have to deal with the prices of the stock that goes down which brings about chaos.
Output: I have to handle the decreasing prices of the stock which causes chaos.
Generally, inputs to our problem are
(a) a set of “elementary” English sentences/writing samples, call this Set E, and
(b) a set of “refined” English sentences/writing samples, call this Set R.
Our model will learn to transform writing samples from Set E into the style of writing samples in Set R.
Output: The study being conducted benefits from past findings.
Output: We went to Barcelona for the weekend. I’ll talk to you later about it.
Output: I have to handle the decreasing prices of the stock which causes chaos.
Relevant Work
CycleGAN by Zhu et. al (March 2017)
The cycle consistency notion for unsupervised intra-language translation is inspired by Zhu et al. (March 2017), who trained adversarial networks for unsupervised image-to-image translations using cycle consistency in their paper ‘Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks’. They achieved this rather effortlessly in the image-to-image model as there are underlying relationships between the domains represented in the dataset, such that the dataset contains different renderings of the same underlying scene for each domain. For instance, the dataset has an image of a zebra in a savanna and an image of a horse in a field. In this example and others, although there is a lack of supervision in the form of paired examples, they exploit supervision by having the domains of the images to be distributed identically. Thus, given a set of images in domain X and a different set in domain Y, they are able to learn a mapping G: X → Y such that the output y’ = G(x) x ∈ X, is indistinguishable from images y ∈ Y by an adversary trained to classify y’ apart from y.
The optimal G thereby translates the domain X to a domain Y’ distributed identically to Y. However, there are infinitely many mappings, G, that the model can learn to generate the same distribution over y’, although these arbitrary mappings may not meaningfully translate input x to output y. The concept of cycle consistency was thus introduced to impose a constraint on the mapping G that could be learned.
With this new concept in mind, as before, given an image in domain X, they transform the image to domain Y using a function G: X → Y, but now, as an added step, this translated image is fed to the inverse function F: Y → X to reconstruct the original image, X. Cycle consistency is basically the workaround to get effective translations of images without having supervision in the form of paired data. To reduce the space of possible mapping functions, the learned mapping functions, F and G should be cycle-consistent.
x → G(x) → F(G(x)) ≈ x, we call this forward cycle consistency.
y → F(y) → G(F(y))≈ y, we call this backward cycle consistency.
Below is a figure from the original CycleGAN paper that helps to illustrate the process.
Our approach draws influence from this paper in that we believe that if cycle-consistency can be used to translate images in an unsupervised way, then we can extend this concept to the Natural Language domain to achieve translations without parallel corpora. In much the same way as the authors of the CycleGAN paper, we can exploit the property that translation should be ‘cycle consistent’, in the sense that if we translate a sentence from English to French and then translate it back from French to English, we should arrive back at the original sentence.
Later in this blog post, we present some of the challenges we have faced in training with cycle consistency. One more major challenge involves exploiting supervision at some level, as in the image domain it was assumed that the images were identically distributed.
Unsupervised Machine Translation Using Monolingual Corpora Only by Lample et al. (October 2017).
The authors consider an encoder and decoder, respectively for encoding source and target sentences to a latent space and for decoding from that latent space to the source or target domain. Initially, the encoder-decoder pair is trained separately on the source and target domains, and the model, thus, learns to denoise sentences in each domain. This is termed as ‘Denoising Auto-encoding’. The autoencoder is then trained on cross-domain to give a noisy translation from source to target and vice-versa. Cross-domain training enables us to translate an input sentence from a source domain X to the target domain Y. To get accurate translations, this architecture uses adversarial training which discriminates between the encoding of the source and target sentences. Adversarial training constraints the encoder to map sentences in the two languages into the same latent space.
This method notably contrasts with the method that we propose in our paper in that instead of using cycle consistency to make the latent vectors consistent for the source encoding and target encoding, this paper uses adversarial training to force the encoder’s latent vector representations for the two domains to be indistinguishable from each other.
Source: Lample et al. The above figure explains the general architecture of the model.
Source: Lample et al. The above figure illustrates denoising auto-encoding and cross-domain training.
x → G(x) → F(G(x)) ≈ x, we call this forward cycle consistency.
y → F(y) → G(F(y))≈ y, we call this backward cycle consistency.
Dual Learning for Machine Translation by He et al. (2016)
We also draw validation of our ideas from Dual Learning for Machine Translation by He et al. (2016). In their paper, the authors proposed two neural translation models, one for each domain. Assuming there are already two well-trained language models, we can train this architecture across domains using reinforcement learning, and we can refine the translations by verifying them against the language models. Language models take a natural sentence as the input and output a real value to indicate their confidence that a sentence is a natural sentence in a given language. Analogously, in our system, the two neural translation models correspond to the two textual reconstructing autoencoders, and the well-trained language model is the shared latent space with enforced cycle consistency which refines the translation.
Dual Learning for Machine Translation by He et al. (2016)
We also draw validation of our ideas from Dual Learning for Machine Translation by He et al. (2016). In their paper, the authors proposed two neural translation models, one for each domain. Assuming there are already two well-trained language models, we can train this architecture across domains using reinforcement learning, and we can refine the translations by verifying them against the language models. Language models take a natural sentence as the input and output a real value to indicate their confidence that a sentence is a natural sentence in a given language. Analogously, in our system, the two neural translation models correspond to the two textual reconstructing autoencoders, and the well-trained language model is the shared latent space with enforced cycle consistency which refines the translation.
Architecture Specification
Our model consists of two autoencoders, both of which are bi-directional deep multi-layer RNNs using LSTMs as recurrent units. Each autoencoder consists of two recurrent neural networks - the encoder takes in the input sentences and embeds them into the latent space while the decoder processes the latent space representation to reconstruct the original sentence.
Figure showing the architecture of each autoencoder in our system. Reference figure taken from TensorFlow’s NMT sequence-to-sequence model architecture
After both the autoencoders have been trained, we want to translate sentences from Elementary English to Refined English by feeding the outputs of the Elementary English Encoder to the Refined English Decoder. As discussed in the CycleGAN paper, we can learn an accurate mapping from an Elementary English sentence to a Refined English sentence by using cycle consistency to constrain the encoding produced by our Elementary English encoder. By defining reconstruction loss and forward/backward cyclic loss, we ensure that the encoding that our model produces for a given sentence can be interpreted by either the decoder of the Refined English decoder or the Elementary English decoder. Later in this post, we will provide more details about how we implement cycle consistency for joint training.
Figure showcasing how encoders and decoders of both the Autoencoders are connected to be cycle consistent
Project Plan
In conducting this research, we came up with a series of milestones which we are well on our way to achieving. The list below summarizes what we’ve accomplished and what we are working on currently.
-
[Completed] Literature search: Read about textual reconstruction, grammar correction using deep learning, and neural machine translation
-
[Completed] Refine our approach: read CycleGAN paper and formulate our problem — project proposal submitted
-
[Completed] Find appropriate datasets — settled on one dataset based on Wikipedia
-
[Completed] Build an autoencoder for textual reconstruction
- [Completed] Find a new, larger dataset of refined English sentences and a dataset of elementary English writing samples. No need for parallel corpora
-
[Completed] Train two Autoencoders separately for sentences with grammatical incorrectness and for sentences with correct grammar
-
[In progress…] Add cycle consistency and perform joint training
-
Test and tune model
-
Advanced application: source dataset with more nuanced errors and writing style
-
Report final results
[Completed] Literature search: Read about textual reconstruction, grammar correction using deep learning, and neural machine translation
[Completed] Refine our approach: read CycleGAN paper and formulate our problem — project proposal submitted
[Completed] Find appropriate datasets — settled on one dataset based on Wikipedia
[Completed] Build an autoencoder for textual reconstruction
[Completed] Train two Autoencoders separately for sentences with grammatical incorrectness and for sentences with correct grammar
[In progress…] Add cycle consistency and perform joint training
Test and tune model
Advanced application: source dataset with more nuanced errors and writing style
Report final results
Project Status
Embedding
Vocabulary Size
Google Negative 300 - Word2Vec
3,000,000
GloVe 6B
400,000
GloVe 27B (Twitter)
1,193,513
GloVe 42B
1,917,494
GloVe 840B
2,196,017
Table 1: Vocabulary sizes of different Embeddings
Embedding
|
Vocabulary Size
|
Google Negative 300 - Word2Vec
|
3,000,000
|
GloVe 6B
|
400,000
|
GloVe 27B (Twitter)
|
1,193,513
|
GloVe 42B
|
1,917,494
|
GloVe 840B
|
2,196,017
|
Basic Textual Reconstruction Model
The first model we trained is a textual reconstructor for regular English text. Getting this model to work took significant effort and lots of trial-and-error. We hope that documenting our approach and procedure can save the reader some time if he or she chooses to construct a similar model. Here, we have provided a brief summary of what we tried, our intermediary results, and explained what worked.
In our initial attempts to train the model, we used 300 dimension GloVe 6B embeddings (details in table above) and we used a dataset of around 1,000 sentences extracted from Wikipedia articles. We trained this model for 5 epochs with 20 LSTM Units and we got very poor results from a qualitative and quantitative analysis. On any given input, our model would output “the”, repeated as many times as there were terms in our original input sentence.
After tuning different hyperparameters and trying to train in different settings, we were able to achieve our first non-repetitive outputs when we trained on a larger dataset of around 50,000 sentences from Wikipedia, for 20 epochs with 20 LTSM units, and using the same GloVe 6B embeddings as in the previous model. We learned that to get the autoencoder to work, three things are critical: (1) a large training dataset of sentences, (2) training for more than 5 epochs, which can take several hours on a Google Cloud or AWS instance, and (3) we must preprocess text to deal with punctuation and capitalization. An example output is shown below.
Having achieved non-repetitive outputs given an input sentence, we decided to train two autoencoders in parallel — one autoencoder for Refined English Reconstruction, and another model for Elementary English Reconstruction, which we emulate by using a dataset of tweets.
Refined English Reconstruction Model
The dataset used in the above experiments was extracted out of articles from Wikipedia. The dataset was fairly sized with about 50,000 training sentences, but we wanted to find a new dataset that had two important characteristics: (1) it had more refined, well-written English that was both expressive and concise, and (2) it had an order of magnitude more training samples. In our literature search and in our experiments, we have learned that having a larger dataset can have positive impacts on the accuracy of our model. As such, we found a dataset of 1 million news articles, which we have described in more detail below.
Website with link to download dataset and dataset details: http://research.signalmedia.co/newsir16/signal-dataset.html
The dataset is sourced from 93,000 different news sources and blogs. There are 266k blog articles and 735k news articles. The average length of an article is 405 words. Given this dataset, we extracted and preprocessed the data, removing some sentences, accounting for punctuation, and lower-casing all sentences. This resulted in a final dataset of around 24,910,000 sentences to train on, which is significantly larger than the ~50,000 we had originally.
Additionally, given that the sentences are taken from news articles, we make some assumptions about the quality of the writing, as we are not able to inspect every sentence. There are a few important notes about this dataset that could cause degradations of performance, including the fact that there are many quotes in news articles, and because of the way in which the articles have been scraped, there are many artifacts in the data, including advertisements.
In addition to trying a new dataset, we wanted to expand the vocabulary for encoding and decoding sentences, so we experimented with using different word embeddings. At this point, we decided to try to train a model on the the GloVe 840B embedding and the nearly 25 million sentences. We first had to process the data and the embeddings to create a vocabulary, a training dataset, and a validation dataset. With this work done, we moved to the Google Cloud Platform to train the model.
When we attempted training on the full dataset of nearly 25 million sentences, we ran out of memory on the machine. After investigating the issue and experimenting with swap space and adding RAM, we decided to train on just 1 million sentences for 10 epochs, with 30 LSTM units and 1024 batch size on a Bidirectional RNN.
The best input, output pair of these results is the following:
Input: As a result, jumpshot is able to provide companies with actionable data to improve the ROI of their search marketing campaigns.
Output: As a data is a way to work, with a increase of existing growth from their existing assets on its.
We wanted to improve the model further and we took the following steps:
-
Enhance quality of training data: we pre-processed the data to exclude any sentences that had ‘$’, ‘@’, ‘www’, ‘http’, or numeric characters
-
Increase quantity of training data: use 10 million sentences instead of 1 million sentences
-
Increase training time of the model: trained the model for around 1.5 days, about 12 epochs.
Here are some sample outputs from our final Refined English Reconstruction Model:
Type tokenized sentence: the man went to the mall
Model output: b'the man went to the street eos'
Type tokenized sentence: the dog was missing
Model output: b'the dog was missing , eos'
Type tokenized sentence: plastic is floating in the ocean
Model output: b'unk is plastic in particular surface eos'
Type tokenized sentence: the suspect was later found
Model output: b'the suspect was later found eos'
Type tokenized sentence: the car accident injured three people
Model output: b'the car car unk unk health , eos'
Type tokenized sentence: united nations voted to block russia
Model output: b'unk united voted unk nations to eos'
Type tokenized sentence: we look forward to seeing you
Model output: b'we look to unk you quickly , , eos'
Type tokenized sentence: i have opinions
Model output: b'i have opinions , eos'
Type tokenized sentence: introduce camping to your little ones
Model output: b'unk want to enjoy our steps , eos'
Type tokenized sentence: he had been battling lung cancer
Model output: b'he had been cancer cancer unk lives , eos'
Type tokenized sentence: good morning everyone
Model output: b'good morning everyone , eos'
Type tokenized sentence: to be or not to be
Model output: b'to be or not to be eos'
Type tokenized sentence: to be or not to be that is the question
Model output: b'to be or not to be that is the question , , eos'
Type tokenized sentence: to be or not to be , that is the question
Model output: b'to be or not to be , that is the question , , eos'
Type tokenized sentence: how much does it cost , do you think ?
Model output: b'how unk unk unk doing about it can do not buy , eos'
At this point, we were satisfied with the results of our Refined English Textual Reconstruction.
Enhance quality of training data: we pre-processed the data to exclude any sentences that had ‘$’, ‘@’, ‘www’, ‘http’, or numeric characters
Increase quantity of training data: use 10 million sentences instead of 1 million sentences
Increase training time of the model: trained the model for around 1.5 days, about 12 epochs.
Elementary English Reconstruction Model
To model elementary English we decided to utilize a dataset of tweets, since they’re a good representative of conversations with possible misspellings, colloquialisms, and abbreviations. As the vocabulary of tweets is not restricted to words from the standard English dictionary, we couldn’t use the same GloVe word embeddings, and, instead, we used GloVe Twitter 27B, which has a vocabulary size of more than 1 million terms taken from tweets, which is made available by Stanford’s NLP group. Just switching to the twitter embedding didn’t solve the problem of unidentifiable words and we had to pre-process the tweets and strip them of non-unicode characters and numerals - including hashtags and urls. This helped to reduce the number of unknown words in the input sentences, leading to better performance.
Our Elementary English autoencoder consists of 30 LSTM units and was trained using a batch size of 1024 for 50 epochs. It achieves near perfect reconstruction for phrases but suffers when given uncommon words or long sentences. As we will show in the preliminary results section, it has also mapped semantically similar tweets close to each other in the latent space. Below we have included some sample inputs and outputs from our Elementary English Reconstruction Model.
Textual reconstruction by the Elementary English autoencoder trained on tweets
Preliminary Results
After implementing the autoencoders, we wanted to visualize how their latent spaces relate to each other. In the images below, we visualize the latent 60-dimensional representations of 100 sample sentences by using t-SNE and PCA to reduce their dimensionalities to 2 dimensions. Our intention for creating these visualizations is for them to serve as a baseline to evaluate the effect of joint training on the latent vector representations.
Latent representations for Elementary English Autoencoder
First, we visualize the latent representations of sentences generated by the Elementary English encoder. We can observe that semantically similar sentences are grouped close to each other in the visualization, for example:
-
Sentence 11 - Good Morning everyone,
Sentence 12 - Good Evening everyone
-
Sentence 76 - God bless America,
Sentence 82 - Thanks South America
-
Sentence 80 - When you take it too far
Sentence 87 - When you realize there’s dessert
On the other hand, we notice that dissimilar sentences appear to be far apart in the 2-dimensional visualization of the Elementary English encoder. For example:
-
Sentence 72 - For the first time in his career
Sentence 96 - Taking care of people when they need it most.
-
Sentence 2 - Plastic is floating in the ocean,
Sentence 95 - Freshly squeezed orange juice
-
Sentence 19 - Here is a thread about
Sentence 44 - The New England Patriots made world history
Sentence 11 - Good Morning everyone,
Sentence 12 - Good Evening everyone
Sentence 12 - Good Evening everyone
Sentence 76 - God bless America,
Sentence 82 - Thanks South America
Sentence 82 - Thanks South America
Sentence 80 - When you take it too far
Sentence 87 - When you realize there’s dessert
Sentence 87 - When you realize there’s dessert
Sentence 72 - For the first time in his career
Sentence 96 - Taking care of people when they need it most.
Sentence 96 - Taking care of people when they need it most.
Sentence 2 - Plastic is floating in the ocean,
Sentence 95 - Freshly squeezed orange juice
Sentence 95 - Freshly squeezed orange juice
Sentence 19 - Here is a thread about
Sentence 44 - The New England Patriots made world history
Latent Representations for Refined English Autoencoder
First, we visualize the latent representations of sentences generated by the Refined English encoder. We can observe that semantically similar sentences are grouped close to each other in the visualization, for example:
-
Sentence 9 - He had been battling lung cancer
Sentence 74 - Happy Thanksgiving from me and mine to you and yours
-
Sentence 6 - We look forward to seeing you
Sentence 68 - We are thankful to have amazing fans
-
Sentence 24 - I am going to play baseball
Sentence 25 - I am playing video games
On the other hand, we notice that dissimilar sentences appear to be far apart in the 2-dimensional visualization of the Refined English encoder. For example:
-
Sentence 72 - For the first time in his career
Sentence 96 - Taking care of people when they need it most.
-
Sentence 14 - How much does it cost, do you think?
Sentence 95 - Freshly squeezed orange juice
-
Sentence 30 - What is your favorite song? Sentence 99 - He tried to destroy me
We now proceed to compare the latent space representations of both the autoencoders using both TSNE and PCA to reduce dimensionality and visualize them.
Sentence 9 - He had been battling lung cancer
Sentence 74 - Happy Thanksgiving from me and mine to you and yours
Sentence 74 - Happy Thanksgiving from me and mine to you and yours
Sentence 6 - We look forward to seeing you
Sentence 68 - We are thankful to have amazing fans
Sentence 68 - We are thankful to have amazing fans
Sentence 24 - I am going to play baseball
Sentence 25 - I am playing video games
Sentence 25 - I am playing video games
Sentence 72 - For the first time in his career
Sentence 96 - Taking care of people when they need it most.
Sentence 96 - Taking care of people when they need it most.
Sentence 14 - How much does it cost, do you think?
Sentence 95 - Freshly squeezed orange juice
Sentence 95 - Freshly squeezed orange juice
Sentence 30 - What is your favorite song? Sentence 99 - He tried to destroy me
Latent space representations of both autoencoders using TSNE
TSNE doesn’t always produce similar results on successive runs and also contains hyper-parameters to tune. Hence, to reduce the possibility of missing any shared latent space due to sub-optimal hyper-parameters, we visualized the latent spaces using PCA to check for common embeddings. The below figure visualizes the latent space for both the autoencoders by using PCA to reduce the number of dimensions to 2.
Latent space representations of the autoencoders using PCA
We again observe that there is no shared latent space between the encoders even though the vocabulary and embeddings share some similarity. This can be attributed to the fact that the latent space is not constrained in any meaningful way. By defining cycle consistency loss, in addition to reconstruction loss, we can constrain the latent spaces of the two autoencoders so that they share some similarity in the encodings of input sentences that they produce.
Current Efforts
Applying Cycle Consistency to Natural Language Domains
Our model is inspired by the CycleGAN paper, and the mainstay of our system is to jointly train two autoencoders and to build a cross-lingual bridge between the two latent spaces. Training the autoencoders jointly leads our model to learn an encoding of a given input sentence in such a way that either decoder can decrypt a version of the original sentence.
In our implementation of Cycle Consistency in progress, we want to ensure that the Refined English Autoencoder and Elementary English Autoencoders learn to reconstruct their inputs respectively, while still being able to pair the Elementary English encoder with the Refined English decoder. To do this, we propose a multi-batch training method, where each batch can be used to train a certain phase in the model. We describe the four phases of our training below.
In Phase 1 (batch 1), we calculate the sequence to sequence loss of a sentence passed through Encoder A (Encoder for Refined English) to Decoder A (Decoder for Refined English).
In Phase 2 (batch 2), the loss is calculated by passing the converted sentence from Encoder B to Decoder B.
In Phase 3 (batch 3), we calculate the first cycle loss. It is the reconstruction loss when a sentence is (1) input into Encoder A (Refined English Encoder), then decoded by Decoder B (Decoder for Elementary English). (2) The output of the decoded sentence is passed back into Encoder B (Encoder for Elementary English) and finally decoded by Decoder A. At the end of this process, we measure the reconstruction loss between the original input and the final output, and we back-propagate our losses through the entire cyclic computation.
In Phase 4 (batch 4), we calculate the second cycle loss. It is the reconstruction loss when a sentence is (1) input into Encoder B (Encoder for Elementary English), then decoded by Decoder A (Decoder for Refined English). (2) The output of the decoded sentence is passed back into Encoder A (Refined English Encoder) and finally decoded by Decoder B (Elementary English Decoder). At the end of this process, we measure the reconstruction loss between the original Elementary English sentence and the final output, and we back-propagate our losses through the entire cyclic computation.
We believe this training method can successfully avoid the case where the two networks try to emulate each other by learning common weights for both the autoencoders.
To implement the training mechanism mentioned above, we defined the 4 phases on our system of autoencoders and fed appropriate batches for the training mechanism to be effective.
Over the past two weeks, we have focused our efforts on defining a new computational graph which emulates the batch training process we described above. First, we load the models which were trained by minimizing the reconstruction loss on refined English and elementary English respectively into the same session. Then, in accordance with Phase 3 defined above, we link the elementary English encoder to the refined English decoder. Given outputs to the decoding, to complete the cycle, we link the refined English encoder to the elementary English decoder, and finish by calculating the reconstruction loss of the original elementary English sentence after the cyclic reconstruction.
Our most recent implementation of the computational graph does not work as expected, and we have, thus, not been able start the joint training process. To help us in our initial implementation and to overcome our current issues, we have consulted with other implementations of Cycle Consistency -- including a TensorFlow implementation of the CycleGAN paper -- however, as other implementations deal with different architectures (Generative Adversarial Networks for image generation with Generators and Discriminators) rather than LSTM RNNs for Natural Language generation, we are not able to draw direct solutions from these implementations. We believe that it is possible to execute the training process we have described above, though we have not yet completed its implementation.
Discussion of Future Project Milestones
Our most recent implementation of the computational graph does not work as expected, and we have, thus, not been able start the joint training process. To help us in our initial implementation and to overcome our current issues, we have consulted with other implementations of Cycle Consistency -- including a TensorFlow implementation of the CycleGAN paper -- however, as other implementations deal with different architectures (Generative Adversarial Networks for image generation with Generators and Discriminators) rather than LSTM RNNs for Natural Language generation, we are not able to draw direct solutions from these implementations. We believe that it is possible to execute the training process we have described above, though we have not yet completed its implementation.
Given the ongoing work in our project, we want to complete our implementation of the 4-phased training process that we describe in the section above. Once we have trained a model jointly using cycle consistency, we will visualize the latent space representation of sentences using the Refined English encoder and the Elementary English encoder. If we find that there is a notable change from our baseline to our newly trained model, we will attempt to translate Elementary English sentences to Refined English sentences and continue training for more epochs.
Additionally, depending on our final results, our aim is to tune hyper parameters of our model further (size of the latent space representation of sentences, batch-size, dropout, attention mechanism) and to test our model on different datasets. To speak a little bit about our grander vision for our project, we've listed some potential novel applications of our work.
Additionally, depending on our final results, our aim is to tune hyper parameters of our model further (size of the latent space representation of sentences, batch-size, dropout, attention mechanism) and to test our model on different datasets. To speak a little bit about our grander vision for our project, we've listed some potential novel applications of our work.