The use of Transformers for deciphering technical discourse
- 6 minutes read - 1162 words
Table of Contents
Transformers can be used for Machine Translation, Text competition, Summarizing text, and Paraphrasing text. Experimentating with it may even result in a short story about the startups and the crypto market.
You can do that using a SaaS API like OpenAI , a finished product like Quillbot , or locally using an API like Huggingface Pipeline .
Usually, a text transformation has a clear purpose, like translating text. Quillbot is a Paraphraser allowing you to reformulate text, making it shorter or easier to read. Sometimes that offers surprising insights, like reformulating vague natural language until its true meaning becomes apparent.
What are Transformers?
The Transformer approach is mostly used for text, but is also used for protein research and image generation .
There are so-called model tasks of “large language models” or NLP tasks are at the heart of Transformers; some common examples include:
- Sequence Classification
- Extractive Question Answering
- Masked Language Modeling
- Causal Language Modeling
- Text Generation (text2text)
- Named Entity Recognition
- Summarization
- Translation
- Audio classification
- Automatic speech recognition
- Image classification (image2text)
In this blog post is using “Text Generation” and “Summarization” are used to analyze the behavior of models and how the result can be used for text analysis.
Transformers are also used for Machine Translations, and translating languages; in this case hard to understand texts are used for experimentation.
The Interview about the Metaverse book
The abstract phrase - „The Metaverse“ - did not appear in investor calls until 2021, when Matthew Ball produced a “Metaverse“ explainer . It was unrelated to blockchain technology at the time. It had still not made news in IT or finance publications. Six months later, Facebook renamed itself “Meta.”
Matthew Ball, the author of The Metaverse: And How it Will Revolutionize Everything , was interviewed by the Washington Post - In ‘The Metaverse,’ a leading evangelist shies away from prediction explaining his book.
Using AI to understand the complicated text
The Interview is not only about the book but also provides insights into its context and the author. Also, about the crypto crash after March 2022. During the interview Matthew Ball says:
The trillions of dollars in the crypto ecosystem, in my mind, exceeded the proven value and product market fit of the technology by some order of magnitude.
This is an exciting sentence, and it is unclear what it means; it is like, „I can neither confirm nor deny. “ This formulation is almost (Glomar complete)[https://en.wikipedia.org/wiki/Glomar_response]. Most SEO writing support tools would probably say, „Too complex, please reformulate, “ or „Simplify your text; it is more challenging to read than the text of your competition.“. It is 2022, and we live in the age of AI, able to transform and translate text in many ways.
Quillbot (an online paraphraser) allows you to reiterate simplifying a text multiple times. You paste some text into the textbox, select simplify or shorten, and rephrase the text. Then you copy the result into the input box and repeat the process until the effect appears satisfying. After some transformations, only a short sentence remains:
Crypto's trillions of dollars exceeded its value and product-market fit.
You also could translate it into „Crypto market went bust.“ There is, of course, a reason to obfuscate your thoughts in an interview. The interview was about a book, not about getting the attention of „Crypto enthusiasts.“
Experimenting with Text completion
After extracting the true meaning of the interview snippet, it might be interesting to see what happens if the text is extended using Goose AI’s playground . Always using the same sentence:
Crypto's trillions of dollars exceeded its value and product-market fit.
PreTrained Model used: GPT-NeoX 20B
| Temperature | Result |
|---|---|
| 0.95 | I know many crypto investors tuned out of crypto during the experience while this event unraveled. |
| 0.45 | In other words, the company’s real business was not to make money off of investors, but to make money for the world. |
| 0.25 | The company’s stock has fallen from a high of $38.56 in August to $9.98 on Friday. |
| 0.15 | The company’s market cap is now $5.6 billion, down from $84 billion at its high |
| 0.05 | The company’s stock price has fallen by more than 80% in the past six months. |
| 1.05 | The letters piled up. Equities Founder’s Liu used her meeting in Barcelona to tell the executives that the company faced a cash squeeze. |
A short story
We get a whole range of answers only using temperature, and you can create a story out of it.
Crypto's trillions of dollars exceeded its value and product-market fit.I know many crypto investors tuned out of crypto during the experience while this event unraveled.
In other words, the company’s real business was not to make money off of investors but to make money for the world.
The company’s stock has fallen from a high of $38.56 in August to $9.98 on Friday.
The company’s stock price has fallen by more than 80% in the past six months.
The letters piled up. Equities Founder’s Liu used her meeting in Barcelona to tell the executives that the company faced a cash squeeze.
That’s a short story about the rise and fall of a crypto venture. We even got the name „Liu“, and a place. The only temperature for this experiment was a low temperature less risky/creative text completions .
Conclusions
You can create a short story out of a single sentence just by changing the temperature parameter using GooseAI’s playground. However, using hazardous temperatures results in a more or less uncontrolled result. A paraphraser can be used as a sentence rephraser or to rewrite an essay; however, there is still a need to ensure that the text is correct.
The AI „knows“ not only the content on which it was trained and the most likely words which might follow each other, but it also „knows“ the temperature of words. It might have different outputs depending on the parameters and the model used.
For further experimentation and hashing sentences and words presence_penalty, frequency_penalty, and repetition_penalty are looking interesting besides temperature, using API integration, where the penalty is about how likely the AI will change a topic (an annoying habit of Cleverbot ).