🗣️ Expert Q&A Series: Amjad Masad

Greetings, fellow humans. 👋

This is Not A Bot - the newsletter about AI that was definitely not written by AI. I’m Haroon, founder of AI For Anyone, and today, I'm excited to share another installment of Not A Bot's Expert Q&A series.

Today's guest is Amjad Masad, CEO of Replit, an AI-powered collaborative browser based IDE. Replit is crushing it, having just hit 20 million developers on its platform and launched a bunch of kickass AI features.

During our conversation, Amjad and I discussed:

• Immigrating from Jordan to founding Replit

• How Replit became a leading AI coding company

• How AI will continue to impact coding

• The skills it will take to succeed in the new-age of AI

This was such an enlightening conversation, and I hope you enjoy it as much as I did!

Let's jump into it...

Amjad Masad, CEO of Replit, discusses the future of AI and how to succeed

Haroon: [00:00:00] All right. Awesome. Amjad, welcome to the show. I want to start off by just asking you like a tldr, what is the story that eventually led you to founding Replit and building Replit into the company it is today?

Amjad: Yeah, when I was in school back in Jordan, where I'm from, It was it was I didn't have a laptop every time I'd go to class to take a object oriented programming in Java or distributed systems and C, or whatever class it is. , the first step is a series of mundane IT tasks.

Amjad: As many of your audience are developers. They know how annoying it is to set up the development environment, but also maintain it over time. And so I started dreaming about just like being able to code really quickly. And at the time Google Docs was getting really big and it was like, there should be a Google Docs for code. That was the initial idea. And I like naively started building it. I didn't know how [00:01:00] hard it was gonna be. But, turns out sometimes naivety is good because we had an early breakthrough. And my friends and I were the first to leverage Emscripten, which is the technology t hat is, that came before what's called Wasm now, an ability to compile all of different languages to the browser. And we had this big breakthrough and we built what was initially just like a browser based sandbox for coding. And so that was the first in the world, and that sort of put me on the map and got me to the US where I worked at Code Academy for a while and then Facebook, and then started Replit the company in 2016 although it's been a side project for so long.

Haroon: Awesome. And I know there was a big educational focus towards the beginning stages of Replit. Can you describe like what the motivation for that was?

Amjad: Yeah, one of the, fundamental observations that, that I've had early on was that There isn't really that much of a difference between a student and a developer and a hobbyist, [00:02:00] everyone that codes is just doing the same thing. And with the Replit wanted to be designed so that it's easy for students and it's advanced enough for professionals, which is a very hard technical design challenge.

Amjad: When you're learning how to drive, you use the same car, you don't, use a car then throw it away, for example. And so I just thought it was absurd that we had such a, this like clear, bright divide between this is a learning tool versus this is a building tool. And I thought programmers are learning all the time.

Amjad: We learn new languages, new technologies. So really want to blur the line between. Learning and building and the initial product market fit came in the education sector. And that's who was willing to pay us at the time. And so we start, we built a small education business, semi bootstrapped initially.

Haroon: And in terms of just creating a tool that is accessible for folks that are professional developers for [00:03:00] students, the direction that Replit is heading in, it's just shipping AI features left and right. It's awesome to see. When did that start? What was the inception for the automated functionality in Replit?

Haroon: Was that something that was always part of the plan, or was it like, holy shit, GPT-3 is really good, let's start to use it, and then it snowballed from there?

Amjad: Yeah. I mean it, it wasn't that because if it was that, then a lot more people would've be in our position. There's a lot of other developer tools out there that haven't really woken up to the changing fact of how this changes all software. But we all, we knew that because we actually a while back on Twitter, someone tweeted out, someone on our team found the slide deck from 2015 when I was like starting to pitch Replit and in it there's like a master plan.

Amjad: I was inspired by Elon's master plan for Tesla and I had three bullets. And the first bullet was like, we're gonna build this tool that's gonna be really [00:04:00] useful for people learning how to code. And then the second bullet is we're gonna use the network and the network economies that we're gonna build to train machine learning models to make it easy to learn how to code, but also make it better to develop.

Amjad: And we actually had some early prototypes where we did language models, but it wasn't transformer based. They had n-gram based language models which are very basic. We didn't ship them because it wasn't like really that good. But it was very clear to me that especially with deep learning, taking over N L P, I always wanted to try some of the more recent like language modeling techniques.

Amjad: When G P 2 2 2 came out, it was like pretty obvious for us that this is this is the way. . So we started building around that time. We actually had some of the really early demos around GPT-3 coding. The stuff that we're shipping today, we like built in like 2020. There were a lot of restrictions around opening eye and you had to apply for a [00:05:00] freaking license to launch anything.

Amjad: And and then the, some of the economics did not make sense. Ultimately, we ended up d oing our own thing. With the initial version of Ghost Rider, now we use a combination of in-house build technology and open ai. But it was always part of our vision and we're just getting started. I think the, like data that we're collecting at Replit, we haven't really sort of, um, we have some training jobs as we are speaking, running, and really we're just getting started.

Haroon: in terms of the AI capabilities that Replit has baked in, so you mentioned ghost Writer, which I think just was it yesterday or the day before you or sometime this week you released the Ghost Writer chat? So in a way to interact with a chatbot, and I know the new bounties on Replit, I believe part of the AI the augmentation there is that when somebody signs up for a bounty, AI takes the first pass at it. And correct me if I'm wrong,

Amjad: Yeah, we haven't launched that yet. But yeah, that was that's the idea, like ultimately, Replit is about [00:06:00] creating software, right? And we want to reduce the distance between having an idea and creating a piece of software in the world. That's really our North Star metric is how long does it take and how much effort it is, how many kilojoules it takes for a human to go from like an idea to a piece of software.

Amjad: And part of the reason we build bounties is that we realize that like some people are just not gonna wanna learn how to code and they also have money. And so we can just connect them with people who who are coding. And then also, in the future have the AI take the first pass. At the, at, what they want to build.

Amjad: And then maybe they'll iterate with a human on it, or AI takes a first pass and then passes it to, to the, to a human to kind of finish it. But we think that AI in the loop there will probably just reduce the distance between idea and a product.

Haroon: And is the idea that there's also RLHF built [00:07:00] in there as well? Once somebody starts working on the bounty and providing feedback as to how to complete the problem, or correct any mistakes that

Amjad: Yeah the space is moving really fast. Yeah, so right now the way you train these language models is you have a pre-trained layer, which is throw everything the kitchen sink at it, and then there's the supervised, fine tuning so that this is like labeled data.

Amjad: And then there's the reinforcement learning from human feedback. Like even before the R L F, you can actually do a lot with just supervised labeled training. So I'm not sure what thing will look like. There's probably some RL aspect of it. But and I think this is like the state of the art will just improve.

Amjad: There's a lot of work going into like G P t, like 2019 to 2023. The story has been or like to 2022. The story has been scale scale. I think the story's now changing from, okay, we have these really [00:08:00] powerful things, how to make them more useful. And so there's a lot of work into the algorithms, the prompting, the fine tuning, the distillation, all that stuff.

Amjad: So we're watching the space. But we we haven't really started like training on BCI data yet.

Haroon: And in terms of the relationship between programmers and software moving forward where do you think we're gonna land in, let's say 10 years? Of 10 years sounds like forever especially at the rate at which things are progressing. But in 10 years, what does it look like?

Haroon: How do we interact with coding tools, for example, and how do we create software?

Amjad: Fred Brooks wrote a book called The Mythical Man month, maybe in the sixties or seventies. It was the wake of a massive software, project failure. I think at IBM it was OS two. They worked on it for 10 years. They failed to ship and it was this big crisis moment in the world of software. It was like, oh, we don't know how to make software.

Amjad: And his book, one of the seminal books on the [00:09:00] subject where it talked about. Like from the title, you can't throw people at the problem, the mythical man month. You can't a man in a month are not, or a human in a month are not interchangeable. And but in, in that book, there's an essay about accidental complexity and he talks about how, and software, like the real essence of building software is actually very small.

Amjad: A lot of what we deal with in software is what's called accidental complexity, and that really, that asset really resonated with me and it inspired a lot of what I do in my career is that once you see it, that perhaps the minority of our time as developers are actually dealing with customer problems.

Amjad: We're dealing with machine problems. We're, dealing with issues surrounding how do we. Write the code and how do we manage the dependencies and how do we manage the environments? How do we ship the code? All that stuff is accidental complexity. What I think AI is currently [00:10:00] doing best at is like automating a lot of the menial tasks.

Amjad: So I think co-pilot and Ghostwriter in their original form, were ultimately like automating away typing. because you still had to have the thoughts, you still had to have the program structure. You still had to have all these things. Go start a chat. It's a little different in that it actually can give you ideas and can brainstorm with you and can be more like a of a collaborator with you.

Amjad: It can generate tests for you can like refactor things for you. And but most of these things are menial tasks, accidental complexity type tasks.

Accidental complexity

Amjad: I think where we're headed is that AI will eat away at most of the accidental complexity. And I think what will be left at the limit is the customer problems.

Amjad: It's like people building software will be the kind of people that are actually best at finding out problems in the world and [00:11:00] going out and solving it. So I think it will really benefit the entrepreneur. So I think like software development in 10 years is gonna be mostly entrepreneurship because that's what's left after you take away most of the complexity.

Amjad: That being said, I think and I call this a bimodal distribution, where like on the end, one end you have the product focused entrepreneur that we just talked about. I think on the other end, the programmer that's writing the AI code, that's writing the platform code, that's writing the really novel systems level code.

Amjad: I think those are gonna be around and perhaps maybe there's less of them cuz they're gonna be leveraged by ai. But that they're gonna be very productive. Those people are gonna be around and I think everything in the middle will just disappear.

Haroon: So do you imagine that in the future we're gonna have a set of super-models that are able to be very general and solve a ton of different problems, or do you see us moving towards a world where there's just a lot of specific foundational models that are for a very [00:12:00] targeted task?

Amjad: Ultimately that's like the bet on AGI versus engineering the systems for maximum usefulness for a given product or category. My bias would be the latter. That is still gonna be largely an engineering problem. Ultimately I don't think anyone should plan for AGI because it really is this moment of singularity.

Amjad: But in lieu of that, I don't think like massive models like make a ton of sense. Most of the time you'd want to sort of engineer systems that are efficient and cheap and like really large models are none of that. They're also like way over open-ended and what you'd want is like more focused models like when you're running a a job. When you're running an inference job through a super massive model, it's this single feet forward process and you have to run all this compute for any given request. And so that, that makes it incredibly inefficient. And so a lot of these [00:13:00] weights or a lot of neural paths are not actually related to the task that you're trying to do. And there's a lot of emergent things around distillation, around pruning around all these things to take foundation models and make them more application specific. And I think that's really the future.

Haroon: Super cool. I was chatting with May Habib from Writer.com just yesterday, and they had just come out with their own sort of foundational model for specifically enterprise copywriting. And I thought it was really interesting and thought that kind of looked like what the future might look like and a lot of companies releasing those types of things.

Haroon: Really in, interesting to hear your take on it. . And you mentioned. That you don't recommend anybody prepare for agi. A lot of the folks that subscribe to not a bot, they're very interested in how they can prepare for the future, and they're very uncertain. They're getting conflicting opinions about what the future is gonna look like.

Haroon: Do you have any recommendations for folks, let's say in school, young [00:14:00] professionals who are entering a world that's just moving so incredibly quickly? That are a bit scared and just wanna make sure that they're as prepared as possible, learning the right skills so that they're future proof in a sense.

Amjad discusses AGI

Amjad: I wouldn't prepare for AGI in the same way that I wouldn't prepare for the end of days. It's effectively the end of days if the vision of AGI that some of these companies have comes to bear because it's called a singularity moment because you can't really predict what happens after that.

Amjad: And so like, how would you even prepare for that? And you wanna prepare for the more likely world and the world that you can actually predict and the world that is more likely the world that you can potentially predict is a world where, . Yes, there's like massive improvements of technology and there's like insane compounding effects of technology and it's pretty hard to keep up, but it is possible to keep up and understand where sort of things are headed.

Amjad: And I think this is one thing that I've I think leveraged in my career. [00:15:00] When I was first coming up and just a professional software involvement, I took a huge bet on JavaScript, for example. And the reason is because I remember seeing Chrome and v8. I was like, okay, V8 just making JavaScripts like a real language.

Amjad: And now all you know, , all these applications are gonna be web applications. And so I became an expert in web applications, single page applications, and I was contributed to some early GI stuff and then was later part of the ReactJS team and React Native and and I like when I first got into it, everyone thought everyone, if you said in 2008, if you told people you're like JavaScript programmer, they would say, oh, you mean you're like a web designer.

Amjad: That was the kind of what a JavaScript programmer was. Because we have programmers wrote php if you wanna do, web development and then Node came out in 20 10, 20 11, and now you're a JavaScript developer. You can write back in and front end and you can become a full select developer.

Amjad: And so rode this massive [00:16:00] sort of technology tailwind. And I think you wanna find these things and you wanna ride them. Look it, it's obvious that large language models are gonna be a things a thing. It's obvious that transformer models are gonna be huge, a big part of our future. But who knows, something else might get invented pretty soon.

Keep your ear to the ground

Amjad: You wanna keep your ear to the ground. You don't have a opinion about, where things are headed and how you wanna steer your career. It's important not to panic and not to listen to a lot of people who panic or listen to a hype and pessimism and all that. Although it's fun. I engage in hype and sometimes just trying to speculate with the future.

Amjad: But ultimately it's possible to figure out where technology. As as had, it just keeps your, keep your eye open, keep studying, keep thinking about the future.

Second answer

Amjad: I think it's like an interesting question, young people coming up now like how do they really react to this technology? I think being a generalist is more important than ever [00:17:00] and being entrepreneurial.

Amjad: I think the world when it's experienced with massive change and massive technology shifts being adaptable is gonna be super important. And I think. , like the worst thing that you could do is you can be a conformist and you can just like you follow the path. Cuz like most parents and most teachers and most adults that wanna set their like kids up for success, they're gonna have an outdated view of the world.

Amjad: So actually if you listen to them, they're gonna put you on the wrong path. And so you almost want to try to be a little more independent minded and it's more important than ever to be a polymath and to be able to do more things as opposed to like really specialize in one thing.

Haroon: What does that look like in the future? Is it gonna be that you're gonna have a whole host of AI tools available to you to augment whatever task it is that you wanna do and you just the person who's able to just leverage whatever tool there is to figure something out very quickly [00:18:00] and then just expand on it.

Haroon: And I know you mentioned on a podcast interview you did recently, the idea of the ideas guy and the return of the ideas guy.

Amjad: Yeah I, and I think that's right. I think that's where I'm coming from is that I think there's gonna be a lot more returns for creativity and high throughput of ideas. I think people are able to generate like a lot of ideas and are resourceful enough to learn and find new tools are gonna be able to be really massively successful.

Amjad: And I think. Single entrepreneurs, like solo entrepreneurs are gonna be able to leverage a lot of different tools, a lot of AIs to create really potentially big businesses. And being adaptable and being able to generate ideas and execute on them really quickly is probably the most important skill for the next decade.

And that does it for today's interview. Huge thanks to Amjad for joining us and sharing her wisdom.

For more Q&As from leaders in AI, like Mark Cuban, CEO of Runway ML, and co-founder of Hugging Face (coming soon) plus daily AI news, be sure to subscribe to Not A Bot, the world's most subscribed daily AI newsletter. 👇

As always, thanks for reading, and see you next time. ✌️

- Haroon - (definitely) Not A Robot and @haroonchoudery on Twitter