How Things Occurred

Last Friday afternoon, on the 17th of November, Sam Altman’s firing from OpenAI was publicly announced on this blog post by OpenAI. Especially in the post, we understand that the reason Sam Altman was fired was that he did not align with OpenAI’s values and business. The following quotes from the post imply that there was a conflict between the board’s vision for safe and open AI and Altman’s priorities. In some way, they did not like how he was hiding some aspect of his personal financial return at the expense of the company’s vision. Additionally, before Altman’s firing from OpenAI, his speeches all around the world were not aligned with the vision of “AI for research”.

“She brings … understanding of the company’s values, operations, and business … including her experience in AI governance and policy.”

“OpenAI was deliberately structured to advance our mission: to ensure that artificial general intelligence benefits all humanity. The board remains fully committed to serving this mission.”

“The majority of the board is independent, and the independent directors do not hold equity in OpenAI. While the company has experienced dramatic growth, it remains the fundamental governance responsibility of the board to advance OpenAI’s mission and preserve the principles of its Charter.”

Brockman Quits

After Altman’s firing from OpenAI on November 18, Greg Brockman, the co-founder and president of OpenAI, announced his resignation from the company. After that, three senior OpenAI researchers resigned after Brockman, including the director of research, Jakub Pachocki, and the head of preparedness, Aleksander Madry.

Investors Pushing for Altman’s Return

Investors—furious at the turn of events—are reportedly exerting pressure on OpenAI’s board to reinstate Altman. Most employees and other stakeholders (Microsoft, venture capitalists, etc.) were left entirely in the dark about Altman’s firing.

Moreover, Altman has been telling investors that he is planning to launch a new venture; it is quite possible that Altman would put everything in the past and, together with the investors and his followers, start his own AI company. Brockman, after resigning from his position at OpenAI, is also expected to join the effort. It is suspected that it is possibly an AI chip startup, which aligns with Altman’s views of for-profit AI.

The Future

Currently, there is no news about Altman’s firing from OpenAI. Everyone—including investors, employees, stakeholders, and us, the users—is waiting for an explanation of this sudden event. The latest news is that Altman and Brockman are in discussions with OpenAI’s board about returning Altman as CEO. With every new piece of information, I will update this post. Until then, wait for Sir Potata and me to use generative AI to create some really good things and also some really stupid things.

For Further Reading

• Sam Altman is leaving OpenAI: A big Reddit thread about Altman’s firing from OpenAI
• whywassamfired.com: A website with the latest news about Altman’s firing
• OpenAI board in discussions with Sam Altman to return as CEO: A post from the Verge about the discussions surrounding Altman’s return as CEO

Well, actually, it didn’t really end with a boom again at least instantly. But there was also no “excess” public hype on the developments either which gave the researchers some room to breathe. As someone who witnessed a small portion of time between the current boom and the 90’s, I can say that this time, the people who marketed the models were more careful about their choice of words. Rightfully so, because everytime something labeled as AI failed to meet expectations, things went south. When the general public heard the word AI, they thought it would overthrow humanity in time like some garbage sci-fi film or novel. As you can see, there was actually so much going on

But AI was more into our lives than ever. Probably the most used AI is the search engines, but nobody labels it as “relevant site finder AI’’ or something but an “engine.” Another instance are social media “algorithms” that serve content (for ex. YT, Insta) or speech recognition present in computers since Apple shipped their consumer computers with it since 1993. And, 2011 comes with Siri being the first virtual assistant who screamed artificial intelligence, but instead was referred to as “intelligent assistant” in the introduction event, never once mentioning the artificial part. This basically is what I meant in the paragraph above. Now when somebody begins “artif-” (Kuyta: Artificial foods? Sir Potata: Fuck off you aren’t Bill Gates, but also not incorrect :) ) all attention is drawn immediately. Good to see how far we have come.

Let’s come to how we picked up from last winter. One of the factors behind the AI winters before was the lack of computational power, which must have been felt extreme in the 60’s. Thanks to the short wait in the field, Moore’s Law came to help and fixed that problem. The law states that speed and memory of computers double every year, which is an extremely amount of improvement if you are patient enough. (Kuyta: Some fun trivia: “Doubling a penny every day for 30 days”). It was finally the time to make the first dreams of AI come true. Another factor is the data available. The Internet solved this problem very well, maybe too well for some people. We will discuss it when we move to our discussions about arts. But suddenly, very huge datasets to feed AI were born and sampling plus processing became really easy compared to what it was then. This is what the term Big Data refers to. Another term that has the same significance is Deep Learning. Even though the road to Deep Learning began before 2000’s, it began to be used after its successful results were seen around that time. Kuyta has already talked about these two in his post “How Does AI Work?” Keep in mind that these two terms are the reason for all of the developments that this post will mention.

I had already mentioned Deep Blue when talking about Turing’s speculations about the field. The same year, researchers Sepp Hochreiter and Jürgen Schmidhuber developed LSTM (Long Short Term Memory) which is a type of recurrent neural network (RNN). To give a brief explanation of what it does, RNN’s can store information about an element and update it with new input, making them suitable to analyze data where the order is important. This was a big step for handwriting, speech recognition, and natural language processing.

Continuing with language processing again, one important paper, “A Neural Probabilistic Language Model” dropped in 2003 from 4 researchers, Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Jauvin. This paper is the basis of the capabilities that we have today with computers interacting with language, thanks to the new approaches taken. In this paper, the “curse of dimensionality” is when there is too much discrete data, it is impossible to compare data and define correct parameters for a dataset as big as a language. To overcome this problem the researchers have used the probabilistic relation between words that frequently appear next to each other in a sentence. This was a move from “conditional probabilities” to “distributed representations,” as they have concluded. I want to take a quick break and bring what I understood from here. When I was experimenting with ELIZA it was obvious that the responses were triggered by specific words in the input which was a conditional response. But ChatGPT (You didn’t see this coming, did you? Kuyta: I had a feeling though) recognizes the patterns in speech and finds the most probable words and sequence to reply with, simply guessing what comes next. (Link to the article).

I probably can’t go without mentioning IBM’s Watson winning Jeopardy! against two champions but I have talked about NLP and speech recognition so much already that I want to move on. For the interested, here is a video of how it went: Watson and the Jeopardy! Challenge.

Now moving on, something that will spark a good amount of love in some of you happened in 2009. Three researchers, Rajat Raina, Anand Madhavan and Andrew Y. Ng (Kuyta: Andrew Ng is actually the GOAT in machine learning tutorials) released a paper titled “Large-scale Deep Unsupervised Learning using Graphics Processors.’’ It’s self explanatory, really. The problem: Unsupervised learning takes darn long. Solution: Take advantage of a computer component that is specifically designed for processing parallel data fast instead of the CPU. Nowadays, it is the basic practice to use GPU for AI. Combined with cryptomining, this surely resulted in some fun moments recently for gamers who just wanted to run their latest extremely demanding video game on a brand new GPU (Kuyta: funny how the prices skyrocketed Sir potata: You use a goddamn Macbook dude, why do you even care).

Let’s talk about image generation now. The type of networks used in the popular image generation programs like DALL-E or MidJourney are called GAN (Generative Adversarial Networks). The term was first coined by Ian Goodfellow in the paper titled “Generative Adversarial Nets” published in 2014. Within these networks, there is G, the generative model and D, the discriminative model. The interesting part is here, while it’s generating, it is also trying to fool itself at the same time, creating “adversity” and a product that is almost indistinguishable from reality. However it is important to keep in mind that this type of network has some setbacks. Training such network is not easy for several reasons. Firstly, the network is prone to memorize the dataset and produces almost identical results to the training data. Secondly, even if it produces something new, there can be a lack of diversity in the output. Lastly, training requires tons of computational resources and time. The first two reasons actually pose problems for artists more than it does to users, which is a topic definitely will talk more about in the upcoming posts. Here is the link for the article.

Then pretty recently, in 2017 a paper called “Deep Unsupervised Learning Using Nonequilibrium Thermodynamics” was published by 4 researchers Dickstein, Wells, Maheswaranathan and Gangull. Seriously, the title sounds like a random word salad, at least to me. For now, I’ll only say that this is the article that discovered the technique that is used in the best image generation programs and it is some real black magic. (Wait for the post about my Image Generation with AI post for more details.) Obligatory article link.

And finally we have come to the part where Kuyta has specifically requested me to talk about, LLM’s. He is absolutely right though, these past years of AI history can’t be concluded without mentioning LLM and transformers. Kuyta has already written a post about them but to look at it historically, they are parallel with transformers. So, what are transformers? In 2017, Google researchers released a paper titled “Attention is all you need” and introduced transformers to the world. Transformers are deep learning architectures that use self and multi-head attention mechanisms, which is what the title is referring to. Self attention mechanisms work to measure the importance of other words in the sentence compared to the one at hand. And multi-head attention combines self attention mechanism outputs to capture different relationships in the data. A transformer consists of three components, encoder, decoder and softmax layer. LLM’s are mainly transformer type of neural networks. Machine translation tools are also based on transformers and still are not perfect. This explains the reason why the researchers back in the 1960’s and 70’s were so disappointed, they were dealing with a problem from 40-50 years in future. Link to the article.

For the ones that made it until here, congrats for your patience :D Believe it or not this concludes the History of AI series. Well, this series doesn’t actually align with the main aim of our blog but I wanted to delve in anyways. I think there is a relation between the current problems with AI and how we got here. From now on, we will discuss the current situation of the field rather than the past and look into the impact AI made over our lives, mainly art and creation.

This series was very challenging to write to be honest because the amount of technical details and abstract concepts that lies behind just one discovery was astounding. But good news, the technical part is mostly over. We can finally move onto the main part of the discussion. Not to pass without mentioning, we have some fun content planned ahead, make sure you follow them. Until then, I’m out.

What are quantum computers?

A quantum computer is basically a computer that exploits quantum mechanical phenomena. In short, at very small states, physical matter exhibits properties of both particles and waves, and quantum computing leverages this phenomenon that even classical physics cannot explain. We shall dive into how quantum computers really work.

How do they work?

In a classical computer, the information is stored using bits. They can be in a state of 1 (electric on) or 0 (electric off), which means that they are deterministic and can only be in one of those mentioned states. By contrast, quantum computers work in qubits which can be in both states of 0 and 1 with equal probabilities, which is called a superposition state. When you measure a qubit with equal probabilities to be both 0 and 1, you get the probabilistic output of a single bit (either 0 or 1), very much like Schrödinger’s cat. Quantum computers manipulate these particles in a way to get the desired outcome. 

If we were in a perfect world where you could have perfect qubits, then we would be fine creating infinite qubits without any problem. But we are not in a perfect world. So, it has been proven that creating a qubit is very challenging. There are a couple things that can make quantum computers suffer from quantum decoherence which introduces noise into calculations (remember that a qubit should have an equal probability of being 0 and 1). To reduce quantum decoherence, qubits should be placed far apart from each other because even a qubit can cause noise to a nearby qubit. They also should not have any thermal noise. Which means that they should be kept at very low temperatures of 0.015 K (-273.135 ºC or -459.643 ºF). Imaging operating a quantum computer, liquid nitrogen isn’t even that cold. It’s mind boggling how scientists can get that low of temperatures.

Everyone is talking about LK-99, what’s the deal with it?

Traditionally, quantum computers are made out of superconductors. They are some kind of materials that have very interesting capabilities (watch this great video to get a physical understanding of superconductors). Also traditionally, superconducting materials were only able to superconduct at very low temperatures, around 3 K (-270.15 ºC or -454.27 ºF). Even the highest temperature at which a superconductor can superconduct at 1 atm is around 150K (-123.15 ºC or -189.67 ºF), which is still very very cold. But now, named after its discoverers and its year of discovery (Sukbae Lee and Ji-Hoon Kim, 1999), LK-99 is set to be a superconducting material that can superconduct in room temperatures. To get the importance of this founding, physicists say that if you really made a room-temperature superconductor tomorrow, you would be famous and you would win the Nobel Prize in physics. So, this is a BIG DEAL. While there are a lot of speculations and criticisms of LK-99, scientists around the world have made big progress in recreating the superconducting material. It seems that this superconducting material might actually be real. LK-99 would allow us to have near-room-temperature quantum computers without most of the limitations mentioned. With that we could start using quantum computers in our homes, yey, who wouldn’t want something like that?

What can quantum computers accomplish?

Recent ground-breaking discoveries aside, quantum computers can do things that traditional computers simply cannot (I will talk about AI in a moment). Thanks to having superposition in qubits (having both states 0 and 1 at the same time), we can do calculations in parallel while classical computers can only do calculations in linear. In simple terms, this means that quantum computers can solve problems by considering all the possible solutions at the same time to that problem to find the correct one. For instance, quantum computers could be great at solving the traveling salesman problem, which is simply the problem of finding the shortest total path between different locations. Quantum computers can also crack some low-level cryptography algorithms (by low-level, classical computers still need years of processing power to crack the algorithm). In short, quantum computers can allow us to process information really fast and calculate simultaneously.

Applications of quantum computers in AI

AI and machine learning models now use tons of data to train. They also need to have billions of neurons to generate what we see. For example, OpenAI’s GPT-4 uses nearly 200 billion neurons to process tons of information. Here, quantum computers can help: they can allow us to process information and data much more quickly compared to classical computers. This, then, could be the opportunity for us to test with neurons and change the values of neurons at speeds never before imagined. Quantum computers’ ability to process information is also useful when training AI and machine learning models. As mentioned, OpenAI used nearly 800 petaflops of processing power to train ChatGPT. The reason is simple: they needed that much power to process terabytes and terabytes of data scraped from the web. Quantum computers would allow us to train models in mere seconds thanks to their parallel processing.

Quantum computers’ application in AI has great potential, from training datasets to analyzing generated content. However, setting LK-99 aside, we still have a lot of time ahead of us until we see real-life use cases of quantum computers in AI. In a couple of years, however, we could see more companies and businesses using quantum computers for various purposes. Overall, the future is bright, and I’m looking forward to it.

ChatGPT lost users?

As this blog is a biiiiiiig fan /s of ChatGPT, it is quite funny that it had a decrease in monthly visits to the website in June 2023. At first, it seems that this is a big issue regarding generative AI; however, we should consider the timing of this decrease in traffic. The school year is over, and usage of ChatGPT could have seen a seasonable decrease due to reduced student usage. And it feels like every tool these days now has generative AI integrated into it, so going directly to ChatGPT’s site from OpenAI is simply less needed.

Kuyta: Oh wow, it has really been a while since we posted anything to this blog. Our last post was from May, and looking back, we probably should’ve posted something; even a simple and short article would have been fine. But we didn’t, and it’s fiiine. Also, expect the last part of History of AI from Sir Potata in the following weeks (hopefully).

What are large language models?

Well, they are language models consisting of neural networks with billions of parameters and weights (to learn more, see my post on how AI works). Large language models are trained on really big datasets, like around half a billion tokens (1000 tokens are around 750 words). These datasets consist of unlabeled text, and the model trains on them using self-supervised training. Unlabeled text means that the dataset is raw and it consists of paragraphs and paragraphs of text, and they are not labelled to specify what the overall thing is mentioned in the text. A self-supervised trained model can identify patterns and structures in the thing that it has trained on; in this case, it is raw text. These kinds of large language models emerged around 2018, and they perform very well in different kinds of tasks. (I’m sure that Sir Potata will write about what happened around 2018–2020 and the boom of large language models in her next post, but until then, we shall wait. maybe not for so long).

The first big boom of large language models in 2018 shifted the focus of natural language processing research away from the previous paradigm of training specialised, supervised models for specific tasks. These specific tasks included sentiment analysis, classification, and detecting spam emails. While these specific task models are in use, most of the present large language models are not specialised in certain tasks.

I should also mention that their skill level and what they can accomplish are dependent on the dataset that they are trained on. It is also connected with the amount of resources the model had access to. These resources include the first-and-foremost dataset, then parameter size, computing power, and neuron depth (huh, now I realise that maybe quantum computers can be used to increase the computing power allocated to the training of these models. Maybe I’ll write a post on it; we’ll see, I guess). This means that the power of large language models is not dependent on additional breakthroughs in design or the mathematics behind them.

What can large language models even accomplish?

The working principle of large language models allows them to be veeeery flexible. I mean, they can literally do anything related to raw text and language processing. They only need a prompt to complete it.

Decide whether a Tweet's sentiment is positive, neutral, or negative. 
Tweet: I loved the new Batman movie! 
Sentiment: 

Like above, you give them a prompt that can be completed, and they will complete it with what they get from the prompt. Models can get what the user is saying through natural languages.

You can even fine-tune a model to your liking. It works by giving the model a couple dozen prompt-completion pairs to further train on. Then, you can ask the model to complete a prompt you give according to your fine-tuning.

{"prompt":"Overjoyed with the new iPhone! ->", "completion":" positive"}
{"prompt":"@lakers disappoint for a third straight night https://t.co/38EFe43 ->", "completion":" negative"}

The funny thing is that last September I tried to fine-tune a model for a chatbot. Note that this was waaay before ChatGPT was even a thing. I used one of my friends’ messages on a Discord server to train the data. Aaaand, it was kind of successful. (Oh, btw, they are in Turkish. If you want me to translate them to English, you can comment on this post to let me know about it.)

Most used large language models

GPT-3, 3.5, 4

GPT-3, GPT-3.5, and GPT-4 are developed and trained by OpenAI (I mean, if you don’t know this company, then why are you even here?). They are basically what I mentioned above as examples of large language models. The user provides them with a prompt, and they generate a completion according to it. The Discord bots I made are made using GPT-3. These models are widely used around the globe for different applications, from medicine to customer support.

BardAI

Now let’s talk about chatbots. Chatbots are made using large language models; you put the entire conversation on the prompt, and then you wait for its completion. With the release of ChatGPT, there are so many more chatbots that anyone can toy with. Let’s talk about some of them.

BardAI is one of the newly developed chatbots by Google. So, Google, I am going to be honest; this chatbot is simply trash. It is totally unlike other chatbots, in a bad way. Firstly, this chatbot is very generic. It cannot write interesting things, and it only writes what I can find with a single Google search. It also really cannot describe useful information. When I asked it about quantum computing in simple terms, it listed a Wikipedia article on quantum computers (I mean, it wasn’t literally a Wikipedia article, but you got me). I believe that this is the reason, as this AI chatbot is very new and needs a lot more time to develop. But until then, I am not going to use this chatbot for any useful stuff.

BingAI

Well, I cannot say much about this AI as I use a Mac and you can only access it through Microsoft’s Edge. Which is stupid, I mean, why would you want to do this? This is such a Microsoft thing to do, really. From what I heard, this bot is much better than BardAI, and it can actually do useful stuff. It seems that it will also get an update where you can upload images to incorporate text generation with image classification to increase your productivity. But other than that, I don’t have much to say about this AI.

ChatGPT

Sir Potata:

Kuyta: THE INFAMOUS CHATGPT. Kuyta, take a deep breath. Now, let’s talk about ChatGPT. It was introduced to the world on November 30, 2022, and since then, the world has not been the same as it was before. Basically, this chatbot is based on GPT-3.5 (and GPT-4 for paid users) and, more or less, has the same capacity as them. The main difference between ChatGPT and other GPT models is that ChatGPT can remember what was discussed a few minutes ago. This means that it can have a human-like conversation with anyone. Ok, well, that’s great! What’s the problem with that? The thing is that this chatbot is SO OVERUSED. So many students and journalists use this model to write essays, articles, journals, etc. However, the model is not very accurate on most of the stuff it writes, and it can make significant mistakes and errors. I also believe that if a teacher gives you an essay assignment, you shouldn’t use any AI bots to write it; only you should write it. Your teacher wants you to learn, not just copy a chatbot.

Another point I want to make is that people use these chatbots in situations where you definitely should NOT use a chatbot that can make errors. Around January of this year, a lot of people kind of started using ChatGPT for medical emergencies instead of calling medical support. This is just ridiculous.

I should also mention that it is not just bad for ChatGPT. While it is not as useful as it could be, it is actually very useful as a tool. ChatGPT cannot write you a full 100 lines of code that is perfect for what you want to do, but it can write a half-assed piece of code that you can use to add what you want. It can create code templates to get you started with your project. You can also ask about a topic, and it can explain to you, in simple terms, what it is. These actually make ChatGPT very useful to some extent.

These use cases for these chatbots are actually endless, but we should not use them extensively for literally everything. Like, I should not use ChatGPT to write this post about ChatGPT; it is stupid. I believe that these chatbots have a very bright future, but we all should be veeeeeery careful about what and how we use them.

Jailbreaking

Maybe you have heard about jailbreaking a chatbot, and maybe you have not heard of it. So, let me explain. Jailbreaking is a trick to make a chatbot do what it shouldn’t or cannot do. For instance, when you ask ChatGPT to give you a summary of how to make a bomb or any dangerous stuff, it gives you a generic message on why it cannot give you a summary of how to make a bomb or anything like that. Jailbreaking in this situation is to make the chatbot give you a summary of how to make a bomb, or something close to it. Basically, jailbreaking makes the bot answer harmful, offensive, or dangerous questions that the user asks. Because of obvious reasons, I am not going to give detailed instructions on how to jailbreak an AI, but all around, you can do this by creating a long storyline that the chatbot is in and overly complicating things so that the detection software does not detect that what you are asking is harmful. OpenAI, or AI companies, try to fix this problem by adding special cases of jailbreaking prompts to the dataset, but the thing is that every time a prompt happens to not work, people can create new prompts to jailbreak the chatbot again. This is simply an endless cycle.

The problem with jailbreaking is that people will, and always will, try to exploit these chatbots. No matter how secure or intelligent it is, people will try to make it do things that it is not supposed to do. With that said, jailbreaking chatbots might also condone people doing bad things to others. And it also violates the laws of AI, if there are any.

In conclusion, I feel that text generating AIs are really really scary when you think about it. And I literally don’t know what will happen tomorrow with large language models, chatbots, and even ChatGPT. Well, actually, I hope that the latter dies. I should mention that this post is the longest post that I have written for the Cookie Blog and I mean, wow. Hope to see you on the next one. Biy!

In the last post, I mentioned some of the improvements that were made but it should be noted that there were much much more than I could bring up. I strongly urge the people who are interested in this topic to do further exploration because there are some pretty neat programs and computer engineering going on. It is amazing how much was achieved with very little computational power compared to what we have now.

The field between 1950 and 1960’s could be described as booming, expanding to every side and it seemed like there were endless possibilities. Everybody had positive views on how the research was going, the hopes were so high that most people believed that AI would reach the level of human thinking in the near future. Then came the 1970’s and everything stopped. The AI field experienced its first recession. This era in AI history is called the First AI Winter (1974-1980). Just as everything was going so well, what happened for it to all come down?

There are several reasons for why it happened. First of all, due to recent developments, the expectations exceeded the abilities of what technology and the methods that were discovered at the time could produce. The lack of memory and operation capacity of computers and the limits it presented was soon to be understood. The main disappointment in AI was actually machine translation. Machine translation, despite how far we have come today, is still a field that is under development. Google Translate, still sucks at translating to some languages. Machine translation was one of the topics that were focused on and was encouraged by the American government at the time. In 1966, the ALPAC (Automatic Language Processing Advisory Committee) released a report about the inconvenience of machine translations which led to the end of fundings on projects.

Mentioning funders, we can’t pass it by without mentioning the amount of funding lost before the winter came. An important funder, DARPA (Defense Advanced Research Projects Agency), cut its support when the results of AI research were seen as unuseful to militaristic goals that they had.

The next blow to the field is the book released by Marvin Minsky and Seymour Papert in 1969. It was titled “Perceptrons”, taking its name after the artificial neural network type. However, this book was not written to praise perceptrons but to criticize it. Perceptorn is a supervised learning algorithm (you can read Kuyta’s post “How Does AI Work” for more info) that classifies inputs based on their weights that represent the importance of a data. One of the problems they pointed out was the inability of perceptors to solve problems that contained XOR function. While this problem was only present in single layered perceptrons, building multi layered perceptrons was not exactly in the scope of researchers. This is how connectionism, the usage of artificial neural networks were abandoned temporarily to be picked up again later.

By 1973 the last blow arrived with the Lighthill Report written by Sir James Lighthill. To summarize its content, it is pessimistic compared to what people thought about AI 10 years ago or so. The report has separated the AI researches in 3 groups titled A (advanced automation), B (bridge activity) and C (computer based CNS research). There is even a section titled “Past Disappointments”. It is not hard to see that this report has led to cuts in fundings in the UK which spread to the rest of the world. One problem that was mentioned in this section is combinatorial explosion, the increasing complexity of a problem with every step. (If you are interested here is the link to the text of report)

The next decade 1980’s was again good times for AI technology. The main focus of AI shifted from searching for what is possible in general sense to producing something useful in a limited area. These programs in general were called the expert systems. These systems were created with cooperation of experts from the areas and were commercialized and implemented into the real world. This type of AI mainly operated on if-then rules. The first system that entered the market was XCON or R1 which basically chose computer components according to customer’s orders. It was a big success with 40 million dollars profit for a year. However, these systems were very hard to maintain and lack of personnel and systems for it to run on made it difficult to develop such systems. Other systems include MYCIN (for diagnosing blood infectious diseases), or PROSPECTOR (for analyzing rock formations). (They know lower case letters do exist, right?)

After the book that buried connectionism 6 ft. under the ground, it somehow found its way back. The name of the hero to bring it out of its grave was backpropagation. David E. Rumelhart who dropped this algorithm in 1985 actually is not the one who invented it (Frank Rosenblatt was the one who did), nevertheless this algorithm was used in recognition algorithms (speech, text) later on. The main idea of backpropagation lies in its ability to set its parameters by propagating the output back in the system and correcting its errors by itself.

Now that AI had returned with practical uses, so did the extreme optimism. And I would like to dramatize this post, saying “oh but nobody would have guessed how the second winter came” etc. But they did. Visiting AAAI’s (American Association of Artificial Intelligence) meeting that was held in 1984, two researchers Roger Schank and Marvin Minsky (I swear this man opens his mouth and it all goes down) pointed out that with the hype revolving around AI would bring disappointment along. And they were right, it came rather quickly, of course again with a cut of fundings (guess who).

One of the main reasons for the revival of AI was the Fifth Generation Computer System project that was funded by Japan’s Ministry of International Trade and Industry. Among the contents of this project was building advanced computers and of course, AI systems. Starting in 1982, the goals set for the project were very high and they were not quite met towards the end of the project. But this project led to worldwide support for AI projects in 1980’s.

With these events shaking the field once again, the funders turned their attention to projects that yielded more immediate results than AI. Also late 1980’s was the time of desktop computers available for general use. The computers developed by IBM and Apple were more accessible and cheaper. What’s more, they could run Lisp (read the first part if you missed it), in some cases even faster than the machines that were built for this job, namely the Lisp machines. Soon, Lisp machines fell out of competition and the market collapsed.

In all, the second winter lasted from 1987 to 1993 and it is the last AI winter that we have experienced so far. The trend is actually pretty noticeable. Something happens in the field that causes the hype, then this hype leads to disappointment and disappointment leads to a winter. This cycle begs the question: “Is the next winter around the corner?” Looking at the state of AI right now, it is easy to say that there is no small amount of hype revolving around it. Don’t take my point seriously now, I’m not an expert or anything but looking at the trend, an AI winter in 10 years is a possible guess. However, this technology is not new anymore and it seems some lessons have been learned during the ups and downs which I will explain in the third part of this series. So it may not happen this time and we may not cast everything aside, halting progress. This time, maybe we will experience a radical revolution altogether that nobody would have guessed.

Now that we got the two AI winters out of the way, recent history of AI is next. That will be all for this post. Thank you for making it through, I will be back with the last part of the series soon. (Seriously, this was supposed to be the last part.)

(Sir Potata: He’s an elitist when it comes to music. STAY AWAY. But check out his music. It’s pretty cool.)

Kuyta: So that’s why I wanted to write this post on music creation. I want to show everyone AI’s music creation.

How does Music Generating Models Work?

In my other post, we learned how machine learning models actually work. Sound-generating models are not an exception to what I mentioned. Basically, we gather music files (like the ones in this dataset) that have many different styles, like classical, electronic, hip-hop, rock, metal, etc. We also should know every single music file’s metadata, like its lyrics, type, genre, and artist who made the music. After that, we can train the machine learning model on the gathered dataset. Some music-generating models will also have a language model attached to them. With the language model, we can write anything that we want to be made, and the music-generating model can make it.

Cool Machine Learning Models for Music Creation

MusicLM

First music generating model that I came across was Google Research Team’s MusicLM. This model is actually one of the newest ones, published on January 26. This model can create a music file from a text description of the music wanted. Let’s look at two of the 30-second pieces that the model created.

As seen from the second example, the model can replicate human singing and humming. While it cannot replicate real lyrics, it can still give the impression that it is saying something in a different language.

The model can also replicate a melody given to it and create a new music from the given melody.

One of the interesting things that this model can do is it can create a whole piece from different text descriptions. It basically can transition from anything to anything without a problem.

Overall, this model is very capable of generating music, and I really like the style that it creates. I should also mention that there isn’t a place where you can try this model for yourself. I took the audio from the paper itself, and there are many more examples of what this model can do in it.

JukeBox

Jukebox was made by the infamous OpenAI. By capturing over 1.2 million songs and their lyrics over the internet, they created this model to generate a song by the given genre, artist, and style. Over the 7000 songs that they created, I chose some of the best ones from the list, and here they are.

BoomyAI

Well… This AI is actually incredible. I am very impressed with it. You just create an account on the platform and start creating music of your own. Basically, you just select your music type, instruments and sounds. Then, you click “create” and voila. You now have created a song. I played with this model for some time and I created many songs. Here are some of my creations.

Beat Blender

Beat Blender is an experiment made by Google engineers to create music with simple beats. From the main beat that you create, the model generates more with four different main styles called “4 corners”.

While the model is not that advanced compared to other ones we analysed, the system that they designed is very simple and useful. Anyone with an interest in music creation can try this model. People who are also interested in machine learning can try to learn how the system actually works.

Runn and Sornting

RUNN and Sornting were made by a single software engineer. The games are actually very different from what we mentioned above. These are not mainly for the music generation. They are actual games. Both of them use machine learning models to generate music, and the game revolves around that generated piece.

RUNN has the infinite-runner type but it is actually not infinite? For me, this game was very, and I mean very, hard to play as a single person. You might actually beat (haha, a pun because the game is about beats) it with two players.

Sornting is a game about placing the missing parts of a song. Unlike the other one, this one is very enjoyable because every time you play, the pieces are different from the last time (because the model creates new music every time). I actually beat the game with zero mistakes. 😎

Ethics of Music Generating Models

It is not all butterflies and rainbows in the music generation, however. Everything that I mentioned is really great, and I actually wonder what will happen next. But I am also concerned about the ethics of creating these models. As mentioned, OpenAI used over 1.2 million songs to train their model, and MusicLM used over 280,000 hours of music to train their model. A high percentage of those pieces were probably used without the consent of the actual creators of the music. Or even worse, they still do not know that their songs were used to train AI. This, firstly, violates the copyrights of the owner. Secondly, this usage is very harmful to the creator, as people can type something like “Taylor Swift style” into the text prompt and get something that has the style of Taylor Swift.

I also fear that the increasing usage of these kinds of machine learning technologies might make newcomers to music composition sad and break their dreams. As seen from the examples by BoomyAI, the AI-generated music is really hard to differentiate from the real ones. This can have big implications in the future if this situation is not dealt with.

(Oh btw, I copied the starting sentence from Hikaru. If you know you know.)

Now that we know about AI’s definition and history, we can talk about how it works. In our other posts, we mentioned how the subfield of artificial intelligence known as machine learning, is widely used in today’s world, and in this post, we will discuss how machine learning models work.

As a recap, if you don’t remember our last post, machine learning is the capability of a machine to imitate intelligent human behaviour. Artificial intelligence systems (machine learning models) are used to perform complex tasks in a way that is similar to how humans solve problems.

Machine learning is one way to use AI. It was defined in the 1950s by AI pioneer Arthur Samuel as “the field of study that gives computers the ability to learn without explicitly being programmed.” This definition is very true today and it is important to understand how machine learning algorithms work. Basically, writing a program that a machine can follow is very time-consuming or straight up impossible. Like, training a computer to recognize human faces. While it is possible to code direct instructions that a machine can follow, it is very very difficult and time-consuming. Here, machine learning takes the approach of letting computers learn to program themselves through experience and trial-and-error.

Now that we know how and why machine learning algorithms are actually and literally learning, we can now get into the nitty-gritty of how machines learn.

Well, as with everything, it all starts with some big data—numbers, photos, or text, like bank transactions, pictures of people, repair records, time series data from sensors, or sales reports. All of this data is gathered and intricately prepared to be used as training data, or the information the machine learning model will be trained on. The more data, the higher the accuracy, and therefore the better the program.

After collecting the big data, which is honestly the hard part because collecting quality data is very, very hard (speaking from experience), programmers choose a machine learning model to use, supply the data, and let the computer model train itself to find patterns or make predictions. This training period can take days or even weeks if your computer is slow, but if you are Google, you can train big machine learning models in seconds (I am totally not jealous). Over time, the programmer can also tweak the model, including changing its parameters, to help push it towards greater accuracy and better results. Also, some data is held out from the training data to be used as evaluation data, which tests how accurate the machine learning model is when it is shown new data.

Now we shall talk about the three subcategories of machine learning.

Supervised machine learning models are trained with labelled data sets, which allow the models to learn and grow more accurate over time. For example, autonomous cars use supervised machine learning models. Humans label hundreds and thousands of pictures with traffic signs, traffic lights, and crosswalks (actually, this is what happened a couple years ago when Google captcha had all of those traffic themed ones, those cheeky bastards), and then the model trains on all of the labelled data and would learn ways to identify pictures of dogs on its own. Supervised machine learning is the most common type used today.

In unsupervised machine learning, a program looks for patterns in unlabeled data. Unsupervised machine learning can find patterns or trends that people aren’t explicitly looking for. For example, an unsupervised machine learning program could look through online sales data and identify different types of clients making purchases.

Reinforcement machine learning trains machines through trial and error to take the best action by establishing a reward system. Reinforcement learning can train models to play games or train autonomous vehicles to drive by telling the machine when it makes the right decisions, which helps it learn over time what actions it should take. Imagine a car feeling pain when it turns the wrong way or when it stops at a traffic light and getting imaginary fruits as a reward. This will make it learn over hundreds of attempts.

Now we can talk about three important models that arise from machine learning algorithms: Natural language processing, neural networks, deep learning, and generative models.

Natural language processing is a research area of machine learning in which machines learn to understand natural language as spoken and written by humans, instead of the data and numbers that are normally used by computers. This allows machines to recognize language, understand it, and respond to it, as well as create new text and translate between languages. They work by simplifying the paragraphs of text in the data set to very simple tokens and training on them. Natural language processing enables familiar technology like the infamous ChatGPT, or Siri, or Alexa.

Neural networks are a commonly used, specific class of machine learning algorithms. Artificial neural networks are modelled on the human brain, in which thousands or millions of processing nodes are interconnected and organised into layers. In an artificial neural network, cells, or nodes, are connected, with each cell processing inputs and producing an output that is sent to other neurons. Labelled data moves through the nodes, or cells, with each cell performing a different function. In a neural network trained to identify whether a picture contains a cat or not, the different nodes would assess the information and arrive at an output that indicates whether a picture features a cat or not.

Deep learning networks are neural networks with many layers. The layered network can process extensive amounts of data and determine the “weight” of each link in the network — for example, in an image recognition system, some layers of the neural network might detect individual features of a face, like eyes, nose, or mouth, while another layer would be able to tell whether those features appear in a way that indicates a face or if it is jumbled mess. Deep learning, like neural networks, is modelled on the way the human brain works and powers many machine learning uses.

Generative models use all of the mentioned machine learning algorithms and tricks to create new images based on a set of instructions (like the famous MidJourney or DallE 2). Generative models normally have two different neural networks: a generator network and a discriminator network. The generator network takes a set of random arbitrary images as input and generates an output image from it. The discriminator network takes an image as input and tries to determine whether it is a real image from the training data set or a generated image from the generator network. While training, the generator network tries to fool the discriminator network into thinking the images that it creates are real images. This adversarial process helps the generator network learn to generate more realistic images over time.

In conclusion, machine learning models are really really complicated and this post just barely scratched the indestructible diamond wall and we still are on top of the iceberg. But, the basics of how machines learning models work is actually this and there ain’t much to say on top of all this. As they say: until next time, I’m out.

Kutay: ChatGPT

Sir Potata:

Aaaaaaanyways, In the past two years or so, AI has become more widely known and easier to use. However, as Kuyta mentioned, it took a lot of time to get to where we are now. Let’s take a look at how AI came to be what it is today and follow its steps throughout history.

I wouldn’t have guessed, but the foundations of AI were laid by people who preceded the technology by at least 200 years. But even before that, people who lived during the ancient times already dreamed of the concept in some way. For example, the bronze giant automaton Talos, the guardian of Crete Island, is based on a Greek myth. While this example is on the imaginative side of things, the logic behind machine learning is also being established. Around 350 BCE, Aristotle introduced the idea of syllogistic logic, a deductive logic system, in his book Prior Analytics. During the 17th century, three mathematicians—Leibniz, Hobbes, and Descartes—worked on expressing thoughts systematically.

Enough with the ancient history; let’s fast forward to times where computers exist, where it all began for real. We are in 1950, Alan Turing published Computing Machinery and Intelligence. This is also where the previously mentioned Turing test is introduced. He discussed the idea of a machine thinking like a human in detail in this paper, including objections and possibilities. He concluded his paper with the following paragraph:

“We may hope that machines will eventually compete with men in all purely intellectual fields. But which are the best ones to start with? Even this is a difficult decision. Many people think that a very abstract activity, like playing chess, would be best. It can also be maintained that it is best to provide the machine with the best sense organs that money can buy and then teach it to understand and speak English.”

The amount of foresight this man possessed was truly astounding. Just two years after he made that statement, Arthur Samuel developed a program that was able to defeat humans in a game of checkers. He was also going to be the one to use the term “machine learning” in 1959. Or the infamous Deep Blue, the computer built by IBM that defeated the world chess champion Kasparov in 1976.

As for understanding the language, that part was achieved to some extent in the 1960s. In 1965, Eliza, a natural language processing program that could interact with the user using language, was created by Joseph Weizenbaum. The concept was rather interesting; Eliza is a psychotherapist. The program used keywords in the answers to come up with an answer. However, the amount of vocabulary was too limited, and the answers were rather cliche. Here is the conversation I had with Eliza for the lols:

Sir Potata: Again, for further explanation of Eliza you can watch this video.

Then there is SHRDLU, another natural language understanding program that could understand, reply, and perform simple tasks using simple sentences. It was developed by Terry Winograd in 1970. (You can see it in action here)

Back in 1956, the historic year for AI, two historic events happened in the field. The “field” previously had no name. John McCarthy coined the term “artificial intelligence” during the Dartmouth Workshop, where the pioneers of AI met up and charted the course for the next decade. Their objective was as follows: “An attempt will be made to find how to make machines use language, form abstractions and concepts, solve kinds of problems now reserved for humans, and improve themselves.” (Here is the full text of the proposal)

The second development was the Logic Theorist by Allen Newell, Herbert A. Simon, and Cliff Shaw. It is officially the first artificial intelligence program in history. The program was used to prove several theorems in the book “Principia Mathematica”. This led to the creation of the Information Programming Language (IPL), which will be important in the creation of AI programming languages.

Kuyta: What? AI programming languages? I’ve actually never heard of that.

Sir Potata: Just like there there are there are specific programming languages specialized for specific tasks, eg. HTML for web programming or python for data analysis, researchers eventually had to come up with an easier way to code AI. As a result of that McCharty came up with the AI programming language Lisp, using IPL as a foundation. Even though it was 1960 when this happened, after more than 60 years, it is still used in the field which if you ask me, is pretty impressive.

What is more impressive is how long this post is becoming. I was a fool for thinking I would be able to fit everything in a post. Obviously it was a mistake. It is fascinating to see something that recently has exploded into popularity to have such history. Next post I will pick up from where I’ve left and complete this topic, until then I’m out.

I will start with what is actually AI. The definition of AI varies with who you ask it to; however, one of the most commonly used definitions of it comes from one of John McCarthy’s papers: “It is the science and engineering of making intelligent machines, especially intelligent computer programs. It is related to the similar task of using computers to understand human intelligence, but AI does not have to confine itself to methods that are biologically observable.”

However, decades before McCarthy’s paper, one of the GOATs of computing, Alan Turing, made a statement about AI: in one of his seminars, he asked the question “Can machines think?” From there, he came up with a test, now famously called “The Turing Test” (how original? (Really, your surname?) This test is made to distinguish between human and computer-generated art, text, or anything really.

The authors of one of the most used textbooks on AI, Stuart Russell and Peter Norvig’s Artificial Intelligence: A Modern Approach, differentiated AI into computer systems on the basis of rationality and humans, and thinking vs. acting.

Computer Systems Based On A Human Approach:

• Systems that think like humans,
• systems that act like humans.

Computer Systems Based on an Ideal Approach:

• Systems that think rationally,
• systems that act rationally.

Most AIs that are commonly used in the world—ChatGPT, Dall E 2, GPT3, and MidJourney—are based on systems that act like humans. Like, for instance, ChatGPT tries to be a human by trying to talk like a human. It tries to have a human conversation with you. By contrast, systems that think like humans are basically trying to learn how humans would think based on a hypothetical situation. For example, a problem solver might not actually solve a problem correctly; however, it will have similar thought processes to real humans who tried to solve the same problem.

As a final word, we can say that AI are computational intelligences that do not have a mind of their own but are things that can mimic human thoughts and actions (I mean, they can mimic reeeally well these days).

With that said, I am going to give the microphone to Sir Potata to talk about AI’s decades-old history.