Author Posts

Max Kanat Alexander understands software implicitly. He has spent his career not only working with it, but thinking about it too. But don't think for a moment that Max is an armchair philosopher. His philosophy has real-world consequences for all programmers, offering a way to write better code and achieve incredible results while remaining happy and healthy in an industry that can be exhausting.  In his new book Understanding Software Max explores a wide range of issues that should matter to anyone working with software - we spoke to him about it, and discussed his thoughts on how thinking about software can help us all.  You're currently working at Google. Could you tell us what that's like and what you're working on at the moment? Max Kanat-Alexander: I can't answer this question in a public setting without approval from Google PR. However, there is a public blog post that describes what I do and provides my title. Why simplicity is so important in software Your last book was called “Code Simplicity” – could you tell us exactly what that means and why you think it’s so important today? MKA: One of the things that I go over in that book, and that I cover also in my new book, is that most of the problems of software fundamentally come from code complexity. Even when it doesn't seem like they do, if you trace down most problems far enough, you'll see that they never would have happened if there hadn't been so much code complexity. This isn't obvious to everybody, though, so I wrote a book that provides a long reasoned argument that explains (a) the fundamental laws of software design and (b) hopefully brings the reader to understanding why simplicity (and maintaining that simplicity) is so important for software. I figured that one of the primary causes of complexity was simply the lack of full and complete understanding by every programmer of what complexity really is, where it comes from, why it's important, and what the simple steps are that you take to handle it. And even now, when I go back to the book myself, I'm always surprised that there are so many answers to the problems I'm facing now. When you've discovered the fundamental laws of software development, it turns out that they do in fact resolve problems even long after their discovery. Do you think you can approach code the same way whatever languages or software you’re using? Is there a philosophy you think any engineer can adopt? Or is flexibility key? MKA: There are fundamental laws and principles that are true across any language. Fundamentally, a language is a way of representing a concept that the programmer has and wants to communicate to a computer. So there are ways of structuring concepts, and then there are ways of structuring things in a language. These both have rules, but the rules and principles of structuring concepts are the senior principles over the rules for structuring things in a language, because the rules for how you organize or handle a concept apply across any language, any computer, any set of tools, etc. Theoretically, there should be an ideal way to represent any particular set of concepts, but I'm not sure that any of our languages are there yet. The philosophy I've expressed in Code Simplicity and now in Understanding Software is entirely a universal philosophy that applies to all software development. I don't generally write about something if you can only use it in one language or with one framework. There is enough of that sort of writing out in the world, and while it's really valuable, I don't feel like it's the most valuable contribution that I personally have to bring to the world of software development. Since I have a background in some types of philosophy as well as a lot of experience doing design (and extensive refactoring) across many languages on many large projects, there's a universal viewpoint that I've tried to bring and that a lot of people have told me has helped them. To answer your last question, when you say the word "flexibility," you're in dangerous territory, because a lot of people interpret that as meaning that you should write endless generic code up front even if that doesn't address the immediate requirements of your system. This leads to a lot of code complexity, particularly in larger systems, so it's not a good idea. But some people interpret the word "flexibility" that way. For a more nuanced view, you kind of have to read all of Code Simplicity and then see some of the newer content in Understanding Software, too. Are there any languages where code simplicity is particularly important? Or any in which it is challenging to keep things simple? MKA: More than for a particular language, I would say it's important for the size and longevity of a project. Like, if I'm writing a one-off script that I'm only going to use for the next five minutes, I'll take a lot of shortcuts. But if I'm working on a multi-million line codebase that is used by millions of people, simplicity is of paramount importance. There are definitely languages in which it's more challenging to keep things simple. I've made this statement before, and there are a lot of amazing developments in the Perl community since I first made it, but it was a lot of work to keep the Bugzilla Project's Perl codebase healthy when I worked on it--more so than with other languages. Some of that had to do with the design of that particular codebase, but also that Perl allows you to accomplish the same thing in so many different ways. It led to a lot more thorough and time-consuming code reviews where we would frequently have to tell people to do things in the way that our codebase does them, or the particular way that we'd learned was the most maintainable through hard-earned experience. It did still result in a well-designed codebase, and you can write well-designed Perl, but it required more language experience and more attention in code review than I've experienced when writing in other languages. Bash is similar in some ways--I once maintained a several-thousand-line Bash program, and that was pretty challenging in terms of simplicity.  Languages aren't equal--some are better than others for some types of tasks. There are tasks for which I love Perl and Bash, and others for which Java, Python, or Go are definitely more suitable. Some of this has to do with the suitability of a particular tool to a particular task, but more of it has to do with things like how much structure the language allows you to place around your code, how much consistency the language libraries have, etc. What makes code bad? What, in your opinion, makes code ‘bad’? And how can we (as engineers and developers) identify it? MKA: It's bad if it's hard to read, understand, or modify. That's the definition of "complex" for code. There's a lot to elaborate on there, but that's the basic idea. What do you think the main causes of ‘bad code’ are? Lack of understanding on the part of the programmer working on the system. A failure of the programmer to take sufficient responsibility for enough of the system. A simple unawareness of the problem. On your blog you talk a lot about how code is primarily ‘human’. Could you elaborate on that and what it means for anyone that works in software?  Sure. It’s people that write software, people that read it, and people that use it. The whole purpose of software is actually to help people, not to help computers or network cables. When you're looking at solving the problems of software development, you have to look at it as a human discipline--something that people do, something that has to do with the mind or the being or the perceptions of individuals. Even though that might sound a bit wishy-washy, it's absolutely true. If you think of it purely as being about the computer, you end up writing compilers and performance tools (which are great) but you don't solve the actual problems that programmers have. Because it's the programmers or the users who have problems--not the machines, for the most part. You have to talk to people, find out what's going on with them. Just as one example, one of the best ways to detect complexity is to ask people for negative emotional reactions to code. "What part of this codebase is the most frustrating?" "What part of this codebase are you the most afraid to touch because you think you'll break something?" Questions like that. And those are basically questions about people, even though you're getting data about code. Because when you come down to it, the whole reason you're getting data about code has something to do with people. You can't lose sight of the fact that the problem you're resolving is code complexity, but at the same time, you also can't forget the reason you're resolving it, which is to help programmers. How has your thinking around code changed over the years? Are there any experiences that stand out as important in shaping your philosophy today? MKA: I've always been fairly meticulous in terms of how I want to approach code. However, it's also important that that meticulousness delivers a product. There have been times, particularly in my early coding career, when I would go to clean up some codebase or improve something only to find that nobody actually wanted the end result. That is, you can work on a project for a long time that nobody actually ends up using or caring about. So one of the lessons that I learned, which is expressed in both Code Simplicity and Understanding Software, is that your software has to actually help somebody or you're not going to actually end up being very happy with it, even if you enjoy the process of working on it. Also, the more time I spend working with groups of engineers as opposed to working alone or in distributed teams, the more I learn about how to communicate the basic principles of software development to other engineers. I think that Code Simplicity did a pretty good job, but weirdly, sometimes you can be too simple for some people. That is, if you get too fundamental with your explanations, then sometimes people don't see how to use the information or how it applies to their life. Sometimes you have to get a bit more complex than the fundamentals, explain a bit more about how to use the data or how it might apply in some specific situation, because some programmers are very focused on their specific problem in present time. That is, they don't want to hear about how to solve all problems like the one they're having - they can't even listen to that or really digest it. So instead you have to frame your explanation in terms of how it applies to this specific problem, which--when you're coming from the fundamental laws of software design all the way up to some extreme complexity that some programmer has boxed themselves into--can become quite difficult to fully communicate. It's also tough because they often think they already know the fundamentals, even though the problem they're having clearly indicates that they don't. So you have to learn how to communicate around those barriers. Working on the Bugzilla Project was a significant influence in how I think about software development, because I proved that you really can refactor a legacy codebase that's gotten into a very bad or very complex state, and that that has significant effects on the product. I think that's one of the things that's been missing in other attempts or writings, is that there's a lot of talk about how code quality is important, but I (and the people who worked with me on Bugzilla) actually went through the process of making significant impacts on products and users over a period of many years by focusing almost purely on code quality and applying the basic principles of software design. There have been lots of other experiences that have been an influence. I think that I generally try to learn something from every encounter I have with programming or software engineers. It's dangerous to think that you already know everything already--then you stop learning. 3 things to take away from Understanding Software What are the three things you want people to take away from Understanding Software? Well, I hope that they take at least as many things away from the book as there are chapters in the book. But in general, I'd like people to have some good ideas about how to handle problems that they run into in terms of code complexity. I'd also like them to understand more about the fundamental reasons that we handle code complexity, as well as the rules around it. I'd like people to see the basic philosophy of software development as something that they can think with themselves, not just something that somebody else has said, and I'd like them to be able to use that basic philosophy to resolve the problems of their software development environments. And of course, I'd like people to come away from the book being much better and faster programmers than they were before.  3 ways engineers can be more productive What 3 tips can you give to developers and engineers who want to be more productive?   1. Understand as much as you can about software, the project you're working on, etc. See the chapter "Why Programmers Suck" in Understanding Software for all the details on this and other things that you should learn about.   2. Try to take responsibility beyond the normal bounds of your code. That is, if you're normally only responsible for one tiny part of the system, also start thinking about how you could improve the parts of the system that talk to your part. Think about the consumers of your code and work to refactor them as well, when necessary. Keep expanding your sphere of responsibility over your codebase over time, even to the point of helping out with the tools and languages that you use to develop software, eventually.   3. Stop thinking so much about your code and start coding more. Or do drawings, or take a walk, or talk to somebody, or something. See the chapter "Stop Thinking: The Secret of Fast Programming" in Understanding Software for the full details on this.

Lorenzo Bettini is an Assistant Professor in Computer Science at the Department of Informatics at the University of Turin, Italy, and the author of Implementing Domain-Specific Languages with Xtext and Xtend. You can learn more about Lorenzo here and here. You can also find him on Twitter: @lorenzo_bettini We spoke to him about his book and his experience of writing it, and found out a little more about his research and daily work… How will readers benefit from this book? Did you learn anything new while writing the book? At the time I started writing the book (and also currently) there was no other book on Xtext (at least in English). For this reason I hope that new users of Xtext can benefit from it: following it chapter by chapter they should be able to get acquainted with this cool and powerful framework. My intention was also to describe my own experiences with Xtext (I've been using it for several years, since version 0.7), in particular I tried to describe some programming techniques and best practices. My two favorite chapters are the one on Testing (Testing your software is truly crucial, and DSLs implemented in Xtext are definitely no exception; the whole book is based on tests) and the one on Scoping (Scoping is one of the most difficult concepts in Xtext, but it is also one of the most important; I hope I managed to describe scoping so that it is easier for readers to understand). For these reasons, I hope that also readers who are already familiar with Xtext can learn something new. Our authors usually have full-time jobs whilst writing for us. Was this the case for you and how did you manage your time? I am a full-time Assistant Professor (Researcher) in Computer Science; this might sound like I have lot of spare time but that's not the case: we too have lots of deadlines… However, since I've always used Xtext for implementing the languages I'm doing research on, the time I spent on the book has been a real scientific investment for me. During the writing process, did you come across any issues/ difficulties that affected your writing and how did you overcome these? Especially for the more advanced chapters I was kind of blocked at least on some example implementation. The authors of the Xtext framework were really available and they helped me solving such issues (not to mention that two of them, Jan Koehnlein and Sebastian Zarnekow, also reviewed the book). I'm really grateful to all of them (especially for creating Xtext) Was there anything interesting that happened during the writing of the book? Well, when I started to write the book, Xtext was version 2.3... After writing half the book, Xtext 2.4 was released. The new release created a new version of the projects (Xtext comes with project wizards that setup most of the things to get you started), in particular, Xtext 2.4 started to rely mostly on Xtend (a Java-like language, completely interoperable with Java and its type system). This meant that all the examples had to be rewritten, and also many parts of the chapters that had been already delivered. I think that this makes the code of the examples (also shown in the book) much more readable, and that's why the title has been changed so that “Xtend” appears in the title as well. How did you find the overall experience of writing your book for Packt? It was a lot of stress, but also a whole lotta fun in the end! What disturbed me most was that I had to use WYSIWYG editors like LibreOffice and Word... I use LaTeX type setting system all the time; LaTeX is so powerful once you learned it that it was a real shock (and nightmare) to fight against the rigidity of Word. What tools or configuration do you use for your workstation? I've been a Linux user for decades, and I've written the book on a very pleasant Linux Mint distribution. I only had to switch to Windows to deal with some problems in the files that required Word instead of LibreOffice. Thanks Lorenzo! If you want to learn more about Xtext and Xtend, you can buy Lorenzo’s book here. Packed with plenty of examples it’s been designed to give readers a practical and accessible insight into a complex area of development.

Claire Chung is a PhD student at the Chinese University of Hong Kong researching bioinformatics. She is also one of the authors of Matplotlib 2.0 by Example. We spoke to Claire about her experience in tech, and her book. She offered a unique perspective on gender issues in the industry as well as wider questions of accessibility and the future of research and how technology will continue to change the way we make scientific discoveries. Packt: What’s the most interesting thing/key take away from your book? Claire Chung: Good storytelling is crucial even for the best data crunching results. Data visualization determines whether you can actually get the message across in presentation. This is true not just for the Tech field, but in any workplace setting - to persuade your clients, your boss, or simply to communicate objective quantitative concepts to fellow scientists or even the general public. Many people around feel scared when it comes to the word ‘code’, or think that numbers alone can provide a complete picture, so sometimes they turn to unsuitable plots that are easily produced with common software, leaving them not customized. We want to tell everyone the right plot and refinement of your data graphics do matter, and it doesn’t take superb programming skills or expensive software to create brilliant data visualization. Packt: Tell us about your experience in the Tech sector, how did you get where you are today and how did you come to write your book with Packt? Did you experience any setbacks along the way? CC: How I went from a bioinformatics lab to writing a Packt book with a lab alumnae today is a bit like joining dots together. Computers have interested me since I was small, because the internet opened up a brave new world of unseen knowledge that was free for me to explore. Moreover, I find huge fun and satisfaction in using logic and creativity in problem solving. I remember persuading my mum at a younger age to let me replace the graphic card based on my own diagnosis, instead of paying double for just a check excluding repair fee, and solved the issue swiftly. Whenever there are technical problems, family, teachers and friends would usually come to me. I enjoy helping them fix the issues, with explanations so they understand how to deal with similar situations. At that time, I did not particularly aspire to be in the Tech field. Sometimes I think that if I wasn’t a scientist, I might have been a dancer or tour guide or anything else. However,  with my curiosity focused on exploring the fascinating deep mysteries of life, I chose to read Cell and Molecular Biology in college. During my undergraduate study, I saw that the Campus Network Support Team in the IT department of my college was recruiting student helpers. I took the opportunity to test self-learnt skills, and gratefully was accepted after the apt-test and interview. While most members naturally come from the Computer Science and Engineering faculty, and were mostly male, I observed a very good work culture there. Everyone was friendly with and willing to learn from each other, regardless of fields of study, education background, gender or anything else. We became good partners at work and good friends in life. Later, I had the privilege to be appointed as leader of the team, responsible for coordinating team effort, recruitment and training of new members, as well as communicating with senior staff and operators in providing support service. This friendly atmosphere and valuable experience play a part in giving me confidence to work in the Tech field and faith that a bias-free friendly environment is totally achievable. In pursuing my studies in life science, I found out that biological data plays a huge role in driving data science. Traditional approaches of genetic research, for instance, study gene actions one by one. Not only is this largely time and labour intensive, we can easily miss the big picture of the network of interactions between genes and different components in the multiple layers of regulations.  Like other Tech fields, technological advancement makes it much easier to generate data, computational power has also increased exponentially in recent decade to power, but data analysis is still at bottleneck. The amount of skilful analysts cannot keep up with the immeasurable data generated each day. While we do not replace traditional studies being of different scope and the basis of research in any scale, there is much more effort needed to innovate for better delineating biology, the workings of life. Packt: How has the industry changed since you started working? Do you think it is getting easier for people from different backgrounds to get into the Tech world? CC: I am working in bioinformatics and have met people in all walks of life getting into Tech. In my point of view, it is certainly evolving into a more open field. Decades ago, programming and data analysis skills used to be vocational expertise. Computer engineering students graduate into programmers in companies. They mostly work in technology-dedicated companies, or are appointed or sent to attach to existing departments on demand. This model still persists, but I have also observed a large paradigm shift nowadays. Companies are hiring more people from diverse background. Startups and new teams in large corporations with high degree of freedom keep emerging. I have friends with art training developing UX award entry apps. I met people with physics, chemistry and civil engineering education and various level of coding background in the same team when speaking in a Tech conference. Today, like how typists have almost disappeared entirely, office productivity software usage is expected as much as basic English writing and arithmetic, basic programming skills are gradually become some sort of literacy. In my case, I am working in a lab among the first established bioinformatics lab in Hong Kong. Not long before I enter the lab, there are so few bioinformaticians, that coming to our lab is one of the very few local options to go when you want to analyse any sequencing data at all. Today we still present an obvious edge in being the most familiar with these techniques, to critically assess and provide solutions to different studies, and hence the most ready to further explore beyond basic analysis, so my supervisor continues to help us select collaboration to get the most interesting questions answered. On the other hand, with increasing availability of courses on bioinformatics and data science, user-friendly packages for data analysis, laboratories are able to hire students who are willing to learn from trial-and-error, or purchase commercial services for basic or routine analyses. While knowledge and techniques set the concrete foundation for anything to build on, I feel that there are no more technical skills that are irreplaceable life-long. Innovation, human-related and overall management skills make the difference. I was really a sidekick at the time I started; there were only two girls including myself. But guess what? We have more or less even males and females joining as regular students now. Packt: What are the biggest misconceptions about working in the Tech sector today? CC: I think the biggest misconceptions are “I am not …, I will never do/understand it”. But more specifically, there seems to be a belief that you must hold a degree in Computer Science or Engineering in order to start your career in the Tech sector today. However, this concept probably isn't as strong in the field as it is outside. Many people outside the field got extremely curious, how I am working in front of computers as a student under a biology programme. While formal major study is undeniably the most direct way to obtain solid foundation of knowledge and to enter the field, it is not a definite ticket to success in the industry. The reverse is true as well. The “Tech field” today is not a shrouded dome. It is a hub where people pass through in getting to their destination. You can start off with a strong computation background, and that can help you land on many different areas. Besides the low-cost learning materials around mentioned, more and more attachment programmes are welcoming aspiring apprentices to bring in new sparks. Think in the other way, starting with less releases you from high opportunity cost. You can also invest & publish your work on GitHub or app stores. Results speak it all. I believe fresher minds on holiday embarking today can probably come up with more brilliant ideas, than I can do with limited time outside my PhD study. We should not be intimidated by things like “I am not white”, “I am not a guy”, “I have not tried this before”, etc. Instead, think of “how you will work it out”. We also should not be bounded by stereotypes against ourselves or anyone else. Personal non-work related qualities and hobbies are irrelevant to ability. (I always wear dress or skirts around in campus or in tech conferences simply because I like it.) Of course, we never need the whole world to work in the same sector, which is not healthy for individuals and society at all. Just don’t set a limit for ourselves. If you want to dive in, be prepared. Then “knock, and the door will be opened.” Packt: What do you think the future looks like for people working in the Tech industry? Will larger companies strive to make it more accessible to everyone? CC: I think the future of the Tech industry will get ever more open and competitive. Will larger companies strive to make it more accessible to everyone? Sure. I think they are doing so even right now. Look at the moves taken at Google cloud platform, Mircosoft STEM resources and NVIDIA’s deep learning institute, just to name a few, including online courses they sponsored. The industry is always in thirst of talent. More open resources bring in more brilliant minds to keep advancement. As discussed, there will no longer be a clear gap set by hours spent or grades obtained in classroom, but willingness to learn and apply actively. Even for people with the strongest formal engineering training, this is the attitude that will push them forward towards far greater success. I am looking forward to seeing more vibrant energy in the field in coming future. Thanks for chatting with us Claire! Check out Matplotlib 2.0 by Example here. Or explore more data analysis resources here.

Heather Mahalik is currently Principle Forensic Scientist and Program Manager at Oceans Edge, Inc, and the course lead for the SANS mobile device and advanced smartphone forensics courses. With over 11 years' experience in digital forensics, she currently focuses on mobile device investigations, forensic course development and instruction, and research on smartphone forensics. As a prolific forensics professional, Heather brought a great deal of expertise and knowledge to Packt, helping to make Practical Mobile Forensics a great success, and we caught up with her to get some thoughts on her experiences as an author. Why did you decide to write with Packt, what convinced you? Packt approached me with the idea and introduced me to the other authors, who ended up being co-authors of the book. I was lucky to be sought out and not have to seek a publisher. As a first-time Packt author, what type of support did you receive to develop your content effectively? Packt provided our team an editor and others to support our efforts on the book. Our Acquisition Editor was fantastic and always responded immediately. I never felt that any question was unanswered or that I didn’t have the support I needed. They were also very flexible with us submitting chapters out of order to allow the normal flow of writing. What were your main aims when you decided to write a book, and how did Packt, in particular, match those aims? I wanted to release a book quickly on mobile forensics that emphasized the use of open source tools. Packt allowed us to progress quickly, update as needed and get the book out. What was the most rewarding part of the writing experience? Working with my co-authors. Seeing their perspectives on each topic was eye opening. What do you see as the next big thing in your field, and what developments are you excited about? Handling smartphone security – device security, encryption, and application obfuscation. Do you have any advice for new authors? Stay positive, write a little bit every day and hang in there. Follow Heather on Twitter (@HeatherMahalik) or take a look at her blog. If you have been inspired to write, get in touch with our experienced team who are always open to discussing and designing potential book ideas with talented people. Click here for more details.

Christoph is the CTO of GESIM, a Swiss start-up company, where he is responsible for their simulation software and web interface built with Angular and D3. He is a passionate, self-taught, software developer and web-enthusiast with more than 7 years’ experience in designing and implementing customer-oriented web-based IT solutions. Curious about new technologies and interested in innovation Christoph immediately started using AngularJS and D3 with the first versions. We caught up with him to get his insights into writing with Packt. Why did you decide to write with Packt, what convinced you? Initially, I wasn’t sure about taking on such a big project. However after doing some research and discussing Packt’s reputation with my University colleagues, I was sure I wanted to go ahead. I was also really passionate about the topic, Angular is one of my favourite tools for frontend JavaScript. As a first-time Packt author, what type of support did you receive to develop your content effectively? I started off working independently, researching papers, developing code for the project and reading other books on similar topics, and I got some great initial feedback from my University colleagues. As the project progressed with Packt, I received a lot of valuable feedback from the technical reviewers and the process really provided a lot of valuable and constructive insights. What were your main aims when you began writing with us, and how did Packt in particular match those aims? I was aiming to help other people get started with an awesome front-end technology stack (Angular and D3). I love to look closely at topics that interest me, and enjoy exploring all the angles, both practical and theoretical, and helping others understand it. My book experience was great and Packt allowed me to explore all the theory and practical concepts that the target reader will find really interesting. What was the most rewarding part of the writing experience? The most rewarding part of writing is getting constructive, critical feedback – particularly readers who leave comments about the book as well as the comments from my reviewers. It was a pleasure to have such skilled, motivated and experienced reviewers on-board who helped me develop the concepts of the book. And of course, holding your own book in your hands after 6 months of hard work is a fantastic feeling. What do you see as the next big thing in your field, and what developments are you excited about? The next big thing will be Angular 2.0 and Typescript 1.5; and this will have a big impact on the JavaScript world. Combining – for example – new Typescript features such as annotations with D3js, opening up a whole new world of writing visualizations using annotations for transitions or styling – which will make the code much cleaner. Do you have any advice for new authors? Proper planning is the key, it will take time to write, draw graphics and develop your code at the same time. Don't cut a chapter because you think you don't have time to write it as you wanted – find the time! And get feedback as soon as possible. Experienced authors and users can give very good tips, advice and critique. You can connect with Christoph here: Github: https://github.com/chaosmail Twitter: https://twitter.com/ChrisiKrnr LinkedIn: https://ch.linkedin.com/in/christophkoerner Blog: http://chaosmail.github.io/ Click here to find out more about Christoph’s book Data Visualization with D3 and AngularJS

IntroductionAPIs are everywhere these days. In fact, they’re responsible for a whopping 73% of internet traffic in 2023, according to the State of API Security in 2024 Report by Imperva. With this level of activity, especially in industries like banking and online retail, securing APIs isn’t just important—it’s essential. The average organization has around 613 API endpoints in production, and as the pressure to deliver faster mounts, that number is only growing. To keep up with this demand while ensuring security, adopting a ‘Shift-left’ approach is crucial. What does that mean? It means integrating security earlier in the development process—right from the design stage, all the way to deployment. By doing this, you’re not just patching up vulnerabilities at the end but embedding security into the very fabric of your API. Getting Started with API Testing API testing plays a huge role in this approach. You’re essentially poking and prodding at the logic layer of your application, checking how it responds, how fast it does so, how accurately it handles data, and whether it can fend off security threats. This is where Postman shines. It’s a widely used tool that’s loved for its ease of use and versatility, making it a perfect fit for your shift-left strategy. With Postman, you can simulate attack scenarios, test for security issues, and validate security measures, all within the same space where you build your APIs. But before we dive into scripting in Postman, let’s get it installed. Installing Postman and Setting Up First things first, if you don’t already have a Postman account, head over to their website to create one. You can sign up using Google if you prefer. Once that’s done, download the version that suits your operating system and get it installed. We’ll need a vulnerable API to test our scripts, and the BreachMe API in my book (API Security For White Hat Hackers) is perfect for this. You can find it here. Follow the documentation to set it up, and don’t forget to import the BreachMe collection into Postman. Just click the import button in the collections tab, and you’re good to go. Postman Scripting Basics Scripts in Postman are where things get really interesting. They allow you to add dynamic behavior to your API requests, mimicking complex workflows and writing test assertions that simulate real-world scenarios. Postman’s sandbox execution environment is written in JavaScript, This means that in order to make a script executable in Postman, it has to be written in Javascript. So, If you’re familiar with Javascript, you’re already halfway there. There are two main types of scripts in postman. The first, pre-request script which is run before a request is rendered to Postman. The second, post-response scripts are scripts that are run after Postman gives a response to a sent request. The order of script execution for a single request is as follows: There are two main types of scripts in Postman: Pre-request Scripts: These run before a request is sent to the API. Post-response Scripts: These kick in after the API responds. The order of script execution for a single request is as follows: Pre-request script You can run these scripts at three levels: the request level, folder level, and collection level. This flexibility means you can apply scripts to individual requests, a group of requests, or even an entire collection. The execution of these scripts will happen in the following order. Dynamic Variables and API Testing During API testing, you often need to work with various user inputs, which can be tedious to create manually each time. One of the coolest features of Postman scripting is the ability to add dynamic behavior to a request. Imagine trying to manually create user inputs for each test—it would be a nightmare. Postman allows you to automate this process, generating variables like random usernames, random IP addresses, email addresses, and passwords on the fly. For example, to generate a dynamic username you can use {{$randomUserName}}. Want a dynamic email? Just use {{$randomEmail}}. And for a password, {{$randomPassword}} has you covered. This makes it easy to send multiple requests to the register endpoint, effectively testing the API.  Postman provides everything and we can now send as many register requests as we need to effectively test the API. Dynamic variables can also be set using pre-request scripts. Post-Response Scripts for Functional Testing Postman can be used to perform essential API testing such as functional testing, this is testing to ensure that the API works/functions in the way it is intended to. When testing functionality, postman allows you to send requests to your API endpoints and validate the responses against expected outcomes. You can check if the API returns the correct data, handles inputs properly, and responds with appropriate status codes (e.g., 200 OK, 404 Not Found). Let’s try that in the above API to check if the login endpoint will return a 200 status code. Navigate to the login endpoint’s script tab and choose the post-response tab.  The script we will use will look like this… Let’s break down the snippet. We’ll use the pm.test() function. The first argument “Status code is 200” will be used as the description of the test. The second argument is a function that will contain the actual test. The pm.response refers to the response object. .to.have.status(200) evaluates whether the response status code is 200.Post-request scripts can be used to set tokens or variables that will be needed throughout the testing of the API. Imagine testing an API and manually copying and pasting variables between requests—tiring, right? This approach ensures the variable is accessible across all requests in the collection, making your testing workflow more efficient, less error-prone, and more secure. Some variables contain sensitive data and may require a bit more protection especially when working in a collaborative environment. Postman recommends using variables in such cases. Let’s take an example of an access token that is short-lived, used by most of the endpoints in the collection, and is set when the login request is successful. To streamline this, we could use a post-response script in the login endpoint to automatically set it. Navigate to the auth folder of the Breachme_API collection and select the login endpoint. Ensure that the username you are trying to log in as is a registered user but using the register login before the login endpoint. When logging in, you’ll require a correct username and password in the body of the request as shown below. The correct credentials will result in a response containing the token. To set it, we will need to get the response; take only the token and set it. The script will look like this: var theResponse =pm.response.json();  pm.collectionVariables.set("access_token", theResponse.token) The first line of code captures the response from the API request and converts it to a JSON object then stores it in a variable theResponse. The pm.response.json() is a Postman function that parses the response body as JSON, making it accessible as a JavaScript object. With the response accessible, we can then get the token using the theResponse.token and set it as a collection variable with the command pm.collectionVariables.set() function. The first parameter will specify the collection variable you want to save it as.  Postman scripts can also be used to validate whether the response contains the expected data. Let’s say you have created a post, you would expect it to have the ‘id’, ‘username’, ‘message’, and maybe an ‘image’. You can use Postman to check if every expected data is returned in the expected format. Let’s check if the register endpoint returns what we expect, with the body {    "username":"user2",    "email":"user2@email.com",    "password":"user2password"  } We expect the response to look like below {    "message": "user created successfully",    "user": {        "id": 2,        "email": "user2@email.com",        "username": "user2",        "is_admin": false,        "password": "#############",        "createdAt": "2024-08-28T22:13:30.000Z"    },    "token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6MiwiZW1haWwiOiJ1c2VyMkBlbWFpbC5jb20iLCJ1c2VybmFtZSI6InVzZXIyIiwiaXNfYWRtaW4iOmZhbHNlLCJwYXNzd29yZCI6IiMjIyMjIyMjIyMjIyMiLCJjcmVhdGVkQXQiOiIyMDI0LTA4LTI4VDIyOjEzOjMwLjAwMFoiLCJpYXQiOjE3MjQ4ODMyMTAsImV4cCI6MTcyNTE0MjQxMH0.Z3fdfRXkePNFoWgX2gSqrTTtOy_AzsnG8yG_wKdnOz4"  } To automate it, we will use the script in the post-response tab of the register endpoint. pm.test("User object has id, email, username, is_admin, password", function () {    const responseData = pm.response.json();    pm.expect(responseData.user).to.have.property("id");    pm.expect(responseData.user).to.have.property("email");    pm.expect(responseData.user).to.have.property("username");    pm.expect(responseData.user).to.have.property("is_admin");    pm.expect(responseData.user).to.have.property("password");  }); To ensure that your API meets performance requirements, you can use a post-response script to measure the response time. The snippet we will use us as seen below: pm.test("Response time is less than 300ms", function () {  pm.expect(pm.response.responseTime).to.be.below(300);  }); The above script uses the pm.test() function with the test description as the first argument and an anonymous function that contains the actual test as the second argument. The pm.expect() function is a Postman function that is used to make assertions, it sets up an expectation for a certain condition. In this case, it expects that the pm.response,responseTime will be below 300 milliseconds. Not meeting this expectation makes the test fail. Conclusion Scripting in Postman isn’t just about convenience—it’s about transforming your testing into a proactive, security-focused process. By using these scripts, you’re not only automating repetitive tasks but also fortifying your API against potential threats. Combine this with other security measures, and you’ll have an API that’s ready to hold its own in the fast-paced world of software development. "As you continue to deepen your understanding of API security and testing, consider exploring "API Security for White Hat Hackers" written by Confidence Staveley. This book is a comprehensive guide that simplifies API security by showing you how to identify and fix vulnerabilities. From emerging threats to best practices, this book helps you defend and safeguard your APIs.Author BioConfidence Staveley is a multi-award-winning cybersecurity leader with a background in software engineering, specializing in application security and cybersecurity strategy. Confidence excels in translating cybersecurity concepts into digestible insights for diverse audiences. Her YouTube series, “API Kitchen,” explains API security using culinary metaphors.nConfidence holds an advanced diploma in software engineering, a bachelor’s degree in IT and business information systems, and a master’s degree in IT management from the University of Bradford, as well as numerous industry certifications such as CISSP, CSSLP, and CCISO. In addition to her advisory roles on many boards, Confidence is the founder of CyberSafe Foundation and MerkleFence.

IntroductionThere is a lot of documentation and some books you can find on the TLS protocol and the theory behind it. But what do you do when you actually have to deploy TLS in a real-world application? For example, what do you do to enable TLS for your web application written in Flask? Securing web applications is more important than ever, and Transport Layer Security (TLS) plays a vital role in protecting data in transit. For developers using Flask, a popular Python web framework, implementing TLS might seem daunting. This article provides a practical guide to setting up TLS for Flask applications, focusing on Nginx as a reverse proxy, secure configurations, and avoiding common pitfalls. Whether you're a beginner or an experienced developer, you'll find actionable insights to ensure your application is both performant and secure.TLS and FlaskIf you want to write a web application, chances are you'll end up using Flask, a popular microweb framework written in Python. Flask is well-known for its simplicity, flexibility, performance, and beginner-friendly learning curve. Flask is a WSGI (Web Server Gateway Interface) application, where WSGI is a standard interface between web servers and Python web applications or frameworks. Finally, a WSGI application is a Python callable that accepts two arguments environment and start_response, and returns an iterable, allowing it to handle HTTP requests and responses in a standardized manner. In Flask, a WSGI server runs the application, converting incoming HTTP requests to the standard WSGI environment, and converting outgoing WSGI responses to HTTP responses. When deploying your Flask application to production, the Flask documentation strongly recommends using a dedicated WSGI server rather than the built-in development server.WSGI servers have HTTP servers built-in. However, when serving your application with a WSGI server, it is good practice — and might be even necessary depending on the desired configuration — to put a dedicated HTTP server in front of it. This so-called “reverse proxy” handles incoming requests, TLS, and other security and performance concerns better than the WSGI server.The Flask documentation describes how to set up Nginx as a reverse proxy. The documentation also provides the following example configuration, but the example does not include TLS support.  server { listen 80; server_name _; location / { proxy_pass http://127.0.0.1:8000/; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-Forwarded-Prefix /; } } So, what do you have to do to enable TLS? In Nginx, TLS and, thus, HTTPS support is provided by a dedicated module called module ngx_http_ssl_module which itself relies on the well-known cryptography library OpenSSL.Here’s an example TLS configuration given in ngx_http_ssl_module documentation: worker_processes auto; http { ... server { listen 443 ssl; keepalive_timeout 70; ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; ssl_ciphers AES128-SHA:AES256-SHA:RC4-SHA:DES-CBC3-SHA:RC4-MD5; ssl_certificate /usr/local/nginx/conf/cert.pem; ssl_certificate_key /usr/local/nginx/conf/cert.key; ssl_session_cache shared:SSL:10m; ssl_session_timeout 10m; ... } But wait, what about insecure TLS configurations? How do you know that your setup will hold off the bad guys?To help you with this task, we wrote a small script which you can download at https://github.com/TLS-Port/TLSAudit. The script reads TLS configuration options in a given Nginx configuration file and prints a warning for any weak or insecure TLS options. We believe the script is helpful because Flask documentation refers also to older OpenSSL versions down to 1.0.2, which reached its official end of life by the end of 2019. These old OpenSSL versions contain algorithms with known cryptographic weaknesses. In the remainder of the article, we want to highlight three important TLS parameters that are checked by the script.TLS CiphersThe ssl_ciphers ciphers directive specifies the enabled ciphers in a format understood by the OpenSSL library, for example:ssl_ciphers ALL:!aNULL:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;The full list of available ciphers can be obtained using the openssl ciphers command. However, not all ciphers supported by OpenSSL are considered secure. Nginx documentation recommends the use of OpenSSL 1.0.2 or higher. Ciphers that should be avoided include:RC4: A stream cipher which is known to be vulnerable for a long time, irrespective of the key length. DES and 3DES: DES is very old cipher whose effective key length is way too short (56 Bit). The 3DES, or TripleDES applies the DES three times. It has a larger key space, but is quite ineffective.MD5-based Cipher Suites: MD5 is a broken hashing algorithm susceptible to collision attacks. Its use weakens the overall security of the whole cipher suite.Export-grade ciphers: Export-grade ciphers -- usually, they have an EXP in their name -- were intentionally weakened (to 40 or 56 bits) to comply with export restrictions from the 1990s. These ciphers are highly vulnerable to brute-force attacks.Early data in TLS 1.3The ssl_early_data directive enables or disables so-called early data in TLS 1.3. If two TLS endpoints share a secret key (this is called a pre-shared key, or PSK), TLS 1.3 allows them to communicate over a secure channel right from the start. This is referred to as zero round-trip time (0-RTT) mode which was added in TLS 1.3. It reduces TLS latency by allowing client Bob to send data to server Alice in the first round-trip, without waiting for Alice’s response. Bob uses the shared key to authenticate the server and to establish a secure channel for the early data which is simply added to the standard 1-RTT handshake. The downside is that 0-RTT data is less protected. First, forward secrecy does not hold for this data because it is encrypted using keys derived from the PSK rather than fresh, randomly generated key shares. This means that if the PSK gets stolen, earlier recorded TLS session can be decrypted by the attacker.Second, 0-RTT data is not protected against replay attacks, where legitimately encrypted and authenticated data are recorded by an attacker and replayed into the communication channel. Regular TLS data is protected against this type of attack by the server’s random variable. 0-RTT data, in contrast, does not depend on the ServerHello message and therefore lacks fresh randomness from the server. Enabling early data can, therefore, decrease TLS security. Elliptic CurvesThe ssl_ecdh_curve directive specifies one or more so-called elliptic curves used in Elliptic Curve Diffie Hellman ephemeral (ECDHE) key agreement. Unfortunately, there are some elliptic curves in OpenSSL 1.0.2 which are insecure according to today's standards:SECP192R1 (prime192v1 or P-192)SECP224R1 (P-224)SECP160R1 and SECP160K1Brainpool curves with a number less than 256TLS versionThe ssl_protocols [SSLv2] [SSLv3] [TLSv1] [TLSv1.1] [TLSv1.2] [TLSv1.3] directive specifies the TLS versions supported by the reverse proxy.  As already discussed, SSLv2 and SSLv3 versions contain serious known security weaknesses and must not be used. But also for TLS versions 1.0 and 1.1 there are known attacks that were shown to work in practice. So, only TLS versions 1.2 and 1.3 should be used (and the former with care).Naming: SSL or TLS?Let's get things straight. Practically everybody surfing the web today uses web addresses starting with https, which stands for Hypertext Transport Protocol Secure. The Secure part is realized by a cryptographic protocol called Transport Layer Security, or TLS for short. SSL, on the other hand, refers to initial TLS versions dating back to early 1990s.To address security needs of the upcoming e-commerce, Netscape Communications started designing a new protocol they named Secure Sockets Layer (SSL) for establishing a secure channel between a web server and a web browser. SSLv1, the first version of the SSL protocol, had severe security issues and Netscape never released its specification or implementation. In November 1994, Netscape publicly released SSLv2, the second version of the SSL protocol, and integrated into Netscape Navigator 1.1 in March 1995. Soon after its release, however, a number of further security weaknesses were discovered in SSLv2. Learning from these flaws, Netscape re-designed SSL from scratch and released SSLv3 in late 1995.In May 1996, the Internet Engineering Task Force (IETF) formed a working group to standardize an SSL-like protocol. The standardization process took around three years and the TLS 1.0 standard was published as RFC 2246 in January 1999. While TLS 1.0 closely resembled SSLv3, it had no backward compatibility to SSLv3. Almost seven years later, in April 2006, IETF published RFC 4346 defining the successor TLS version 1.1. Publication of RFC 5246 and with it a new version TLS 1.2 followed in 2008. Finally, in August 2018, IETF published RFC 8446 which specifies the newest TLS version, TLS 1.3.ConclusionSecuring your Flask application with TLS is essential for protecting sensitive data and building user trust. By following the best practices outlined in this article—such as configuring strong ciphers, enabling secure TLS versions, and leveraging tools like TLSAudit—you can ensure a robust, production-ready setup. For a deeper dive into the intricacies of TLS and advanced cryptographic concepts, consider exploring the book TLS Cryptography in Depth, by Dr. Paul Duplys, Dr. Roland Schmitz. TLS is the most important and widely used security protocol in the world. This book takes you on a journey through modern cryptography with TLS as the guiding light. It explains all necessary cryptographic primitives and how they are used within TLS. You’ll explore the inner workings of TLS and its design structure.Author BioDr. Paul Duplys is chief expert for cybersecurity at the department for technical strategies and enabling within the Mobility sector of Robert Bosch GmbH, a Tier-1 automotive supplier and manufacturer of industrial, residential, and consumer goods. Previous to this position, he spent over 12 years with Bosch Corporate Research, where he led the security and privacy research program and conducted applied research in various fields of information security. Paul's research interests include security automation, software security, security economics, software engineering, and AI. Paul holds a PhD degree in computer science from the University of Tuebingen, Germany.Dr. Roland Schmitz has been a professor of internet security at the Stuttgart Media University (HdM) since 2001. Prior to joining HdM, from 1995 to 2001, he worked as a research engineer at Deutsche Telekom, with a focus on mobile security and digital signature standardization. At HdM, Roland teaches courses on internet security, system security, security engineering, digital rights management, theoretical computer science, discrete mathematics, and game physics. He has published numerous scientific papers in the fields of internet and multimedia security. Moreover, he has authored and co-authored several books. Roland holds a PhD degree in mathematics from Technical University Braunschweig, Germany.

John O'Nolan took some time out of his busy schedule to talk to Packt Publishing about the meteoric rise of Ghost - the stripped back blogging platform. What were the biggest issues you faced/continue to face in the development phase? Open source development is always a challenge. You’re taking a group of people from entirely different backgrounds, with widely varying skillsets, from all over the world, who’ve never met each other before - and trying to get them to work together in a way which is both functional and cohesive. Organising such a large project tends to attract an even larger group of developers, so managing that whole process is generally one of the biggest challenges which we face on a day to day basis. It’s incredibly rewarding when it all comes together and works correctly, but it takes a lot of time and attention to get right. We’ve learned a tremendous amount about this process in the last 12 months, and I’m sure we’ll learn just as much in the next 12 months. Would you recommend Kickstarter for other software devs? Are there any lessons other open-source developers could take away from it? Crowdfunding is something I’d recommend to anyone - but whether or not you use Kickstarter is a really hard one to answer, and I’ve gone back and forth on it a few times. I think these days my answer is more or less: “It depends” - as there are both advantages and drawbacks to each approach. On the plus side, Kickstarter gives a project a certain degree of credibility, puts in in front of people who might not necassarily have seen it otherwise, and gives you quite a lot of tools to manage the campaign. On the downside, they take quite a large chunk of the money you raise, you have to play by their rules, and the tools for managing the campaign are good - but not great. I think if you have an existing network or audience, independent crowdfuning might be a more compelling proposition. What are your views on the situation with Express.js, and the questions is raises overall about the OpenSource movement? The situation with Express.js was interesting, but realistically, things like this happen in Open Source all the time. It’s not a new paradigm, and to some degree it’s a bit of a storm in a teacup. I don’t think it raises any new questions about the Open Source Sofware movement - in fact in some way it points to the very strength of it. Had Express been a closed project which got sold, everyone would’ve been left high and dry. With Open Source, there is always the option to fork and create a new project if the original branch loses its way. It’s not a perfect solution (it’s often not even a good one), but in the face of the alternative: no solution at all - it’s a significant step in the direction of freedom. What's your biggest hope for the future of Ghost? Mass adoption? An increase in dialog about how we distribute our blog content? Mass adoption is always a hope, of course, but I’m definitely more passionate about quality than quantity. I would rather have 5 respected major news organisations running on Ghost than 10,000 cat blogs. I think that for journalism to continue to remain relevant it needs to continue to be free and open, both in the tools being used as well as in the organisations behind the content being created. I hope that Ghost can move the needle on independent publishing, as opposed to venture-backed click-bait. Besides a different use philosophy (just blogging!) Ghost is notable for its embrace of Node.js. Do you think more CMS systems will start to make the transition to Node.js? Yes and no. I don’t believe many (if any) existing systems are going to start transitioning to Node.js. There are far too many reasons for why it doesn’t make much sense. But I do think that we’re already starting to see a definite transition in the technologies chosen for building new platforms. Node.js is certainly towards the front of the pack, but it’s by no means the only player. There are a great deal of exciting new technologies which are lining up to power the next generation of the web, and I’m pretty excited to see where they all go. With Ghost(Pro), and fairly easy going third-party hosting, Ghost's more accessible than many open-source blogging/CMS platforms. What do you think Ghost offers over and above more packaged blog solutions? The choices right now are beautiful and closed, or clunky and open. We’re trying to combine the best of both worlds and create something really special.

What initially drew you to write your book for Packt Publishing? Trying to bundle all the CryENGINE knowledge and everything we have learned over the past decade working with CryENGINE was something we wanted to do for a long time. With a publisher like Packt we felt confident we could deliver a comprehensive CryENGINE guide. When you began writing what were your main aims? What did you want the book to teach your readers? It was really about taking the readers' CryENGINE skills to the next level. There are plenty of beginner tutorials and articles out there that focus on getting to know the engine or on building simple examples. What was really missing was a book that focused on users who already have the basic experience and know their way around CryENGINE. This is why Mastering CryENGINE covers a lot of advanced topics and contains a lot of tips and tricks useful for people using the engine in a real production environment. What did you enjoy most about the writing process and what was most rewarding about the experience of writing? Seeing the book come together and grow larger and larger over the course of many months really felt good. And of course knowing that somewhere, someone will read it one day and improve their skills with this book is something that just makes you feel good about all the work it takes to finish a book. What parts of the writing process were the most challenging? How did you overcome the The single biggest problem was cutting down the content so that it would all fit within the scope of the book. There were many more chapters, examples, and images we would have liked to add to the book. It is easy to keep on adding more content and have the book lose its focus, so we had to take a lot of care to keep the book streamlined and on topic. Why, in your opinion, is CryENGINE exciting to discover, read, and write about? CryENGINE is one of the most advanced 3D engines out there and it offers so many features and possibilities that it makes it hard to really discover everything. Even after having worked with CryENGINE for ten years I still sometimes discover new things and new best practices I didn't know about. And of course CryENGINE is always evolving. New features are being added and old features optimized. This means there are new things to discover on a regular basis that help you to build your games more efficiently. Why do you think interest in CryENGINE has continued to maintain such popularity? Today it is easier than ever to develop and publish games. You don't need to be a big company anymore to get your game published. CryENGINE gives you the tools to build amazing games in a short amount of time and release them to the public. More and more people are discovering that it is easy and fun to try to put their own games together. I think this is why CryENGINE continues to be that popular. What do you see on the horizon for CryENGINE? This year has been a busy year for Crytek and CryENGINE. A lot of features have been added. I think we will see even more features and optimization to the engine in the next months. Focus points of this development will be improvements to the ease of use of the SDK tools as well as to the pipeline and the steps necessary to actually get a release-ready game. Any tips or stories to share for aspiring/new technical authors? The beginning is always the hardest part. Once you start writing, it starts to flow. Especially with a great publisher by your side there is no need to be afraid of trying.

IntroductionThe field of machine learning (ML) and generative AI has rapidly evolved from its foundational concepts, such as Alan Turing's pioneering work on intelligence, to the sophisticated models and applications we see today. While Turing’s ideas centered on defining and detecting intelligence, modern applications stretch the definition and utility of intelligence in the realm of artificial neural networks, language models, and generative adversarial networks. For software engineers, this evolution presents both opportunities and challenges, from creating intelligent models to harnessing tools that streamline development and deployment processes. This article explores the best practices in machine learning, insights on deploying generative AI in real-world applications, and the emerging tools that software engineers can utilize to maximize efficiency and innovation.Exploring Machine Learning and Generative AI: From Turing’s Legacy to Today's Best Practices When Alan Turing developed his famous Turing test for intelligence, computers, and software were completely different from what we are used to now. I’m certain that Turing did not think about Large Language Models (LLMs), Generative AI (GenAI), Generative Adversarial Networks, or Diffusers. Yet, this test for intelligence is equally useful today as it was at the time when it was developed. Perhaps our understanding of intelligence has evolved since then. We consider intelligence on different levels, for example, at the philosophical level and the computer science level. At the philosophical level, we still try to understand what intelligence really is, how to measure it, and how to replicate it. At the computer science level, we develop new algorithms that can tackle increasingly complex problems, utilize increasingly complex datasets, and provide more complex output. In the following figure, we can see two different solutions to the same problem. On the left-hand side, the solution to the Fibonacci problem uses good old-fashioned programming where the programmer translates the solution into a program. On the right-hand side, we see a machine learning solution – the programmer provides example data and uses an algorithm to find the pattern just to replicate it later.   Figure 1. Fibonacci problem solved with a traditional algorithm (left-hand side) and machine learning’s linear regression (right-hand side). Although the traditional way is slow, it can be mathematically proven to be correct for all numbers, whereas the machine learning algorithm is fast, but we do not know if it renders correct results for all numbers. Although the above is a simple example, it illustrates that the difference between a model and an algorithm is not that great. Essentially, the machine learning model on the right is a complex function that takes an input and produces an output. The same is true for the generative AI models.  Generative AI Generative AI is much more complex than the algorithms used for Fibonacci, but it works in the same way – based on the data it creates new output. Instead of predicting the next Fibonacci number, LLMs predict the next token, and diffusers predict values of new pixels. Whether that is intelligence, I am not qualified to judge. What I am qualified to say is how to use these kinds of models in modern software engineering.  When I wrote the book Machine Learning Infrastructure and Best Practices for Software Engineers1, we could see how powerful ChatGPT 3.5 is. In my profession, software engineers use it to write programs, debug them and even to improve the performance of the programs. I call it being a superprogrammer. Suddenly, when software engineers get these tools, they become team leader for their bots, who support them – these bots are the copilots for the software engineers. But using these tools and models is just the beginning.  Harnessing NPUs and Mini LLMs for Efficient AI Deployment Neural Processing Units (NPUs) have started to become more popular in modern computers, which addresses the challenges with running language models locally, without the access to internet. The local execution reduces latency and reduces security risks of hijacking information when it is sent between the model and the client. However, the NPUs are significantly less powerful than data centers, and therefore we can only use them with small language models – so-called mini-LLMs. An example of such a model is Phi-3-mini model developed by Microsoft2. In addition to NPUs, frameworks like ONNX appeared, which made it possible to quickly interchange models between GPUs and NPUs – you could train the model on a powerful GPU and use it on a small NPU thanks to these frameworks.  Since AI take so much space in modern hardware and software, GeekbenchAI3 is a benchmark suite that allows us to quantify and compare AI capabilities of modern hardware. I strongly recommend to take it for a spin to check what we can do with the hardware that we have at hands. Now, hardware is only as good as the software, and there, we also saw a lot of important improvements.  Ollama and LLM frameworks In my book, I presented the methods and tools to work with generative AI (as well as the classical ML). It’s a solid foundation for designing, developing, testing and deploying AI systems. However, if we want to utilize LLMs without the hassle of setting up the entire environment, we can use frameworks like Ollama4. The Ollama framework seamlessly downloads and deploys LLMs on a local machine if we have enough resources. Once installing the framework, we can type ollama run phi-3 to start a conversation with the model. The framework provides a set of user interfaces, web services and other types of mechanisms needed to construct a fully-fledged machine learning software5.  We can use it locally for all kinds of tasks, e.g., in finance6 . What’s Next: Embracing the Future of AI in Software Engineering As generative AI continues to evolve, its role in software engineering is set to expand in exciting ways. Here are key trends and opportunities that software engineers should focus on to stay ahead of the curve: Mastering AI-Driven Automation: AI will increasingly take over repetitive programming and testing tasks, allowing engineers to focus on more creative and complex problems. Engineers should leverage AI tools like GitHub Copilot and Ollama to automate mundane tasks such as bug fixing, code refactoring, and even performance optimization. Actionable Step: Start integrating AI-driven tools into your development workflow. Experiment with automating unit tests, continuous integration pipelines, or even deployment processes using AI. AI-Enhanced Collaboration: Collaboration with AI systems, or "AI copilots," will be a crucial skill. The future of software engineering will involve not just individual developers using AI tools but entire teams working alongside AI agents that facilitate communication, project management, and code integration. Actionable Step: Learn to delegate tasks to AI copilots and explore collaborative platforms that integrate AI to streamline team efforts. Tools like Microsoft Teams and Github Copilot integrated with AI assistants are a good start. On-device AI and Edge Computing: The rise of NPUs and mini-LLMs signals a shift towards on-device AI processing. This opens opportunities for real-time AI applications in areas with limited connectivity or stringent privacy requirements. Software engineers should explore how to optimize and deploy AI models on edge devices. Actionable Step: Experiment with deploying AI models on edge devices using frameworks like ONNX and test how well they perform on NPUs or embedded systems. To stay competitive and relevant, software engineers need to continuously adapt by learning new AI technologies, refining their workflows with AI assistance, and staying attuned to emerging ethical challenges. Whether by mastering AI automation, optimizing edge deployments, or championing ethical practices, the future belongs to those who embrace AI as both a powerful tool and a collaborative partner. For software engineers ready to dive deeper into the transformative world of machine learning and generative AI, Machine Learning Infrastructure and Best Practices for Software Engineers offers a comprehensive guide packed with practical insights, best practices, and hands-on techniques.ConclusionAs generative AI technologies continue to advance, software engineers are at the forefront of a new era of intelligent and automated development. By understanding and implementing best practices, engineers can leverage these tools to streamline workflows, enhance collaborative capabilities, and push the boundaries of what is possible in software development. Emerging hardware solutions like NPUs, edge computing capabilities, and advanced frameworks are opening new pathways for deploying efficient AI solutions. To remain competitive and innovative, software engineers must adapt to these evolving technologies, integrating AI-driven automation and collaboration into their practices and embracing the future with curiosity and responsibility. This journey not only enhances technical skills but also invites engineers to become leaders in shaping the responsible and creative applications of AI in software engineering.Author BioMiroslaw Staron is a professor of Applied IT at the University of Gothenburg in Sweden with a focus on empirical software engineering, measurement, and machine learning. He is currently editor-in-chief of Information and Software Technology and co-editor of the regular Practitioner’s Digest column of IEEE Software. He has authored books on automotive software architectures, software measurement, and action research. He also leads several projects in AI for software engineering and leads an AI and digitalization theme at Software Center. He has written over 200 journal and conference articles.

David Dossot is a highly experienced software engineer and architect of close to two decades. Armed with knowledge of RabbitMQ gathered since 2009, he has written a fantastic book called RabbitMQ Essentials, a fast-paced tutorial that covers the fundamentals of RabbitMQ when used in Message Queuing. As a renowned developer, we asked David to share with us his approach to and experience with authoring a technology book with Packt. We also asked him for some of his insights and running thoughts on the current state and future development of RabbitMQ and Message Queuing. You can see his answers below and find more information about his book here. Q. What initially attracted you to write your book for Packt Publishing? Packt Publishing approached me with a project for a RabbitMQ book. Since it's a technology I know quite well and appreciate a lot, and because the time was right, I decided to embark upon the adventure. Moreover, having never worked with Packt before, I was curious to experiment with a new publishing workflow. Q. When you began writing, what were your main aims? I wanted to produce a short book that would have a mix of high-level concerns, like architectural principles around messaging, and low-level concerns, like the technical details involved in dealing with RabbitMQ. I also wanted to make the book easy to read by telling the story of a fictitious company discovering RabbitMQ and rolling it out up to production. Q. What did you enjoy most and what was most rewarding about the experience of writing? I really enjoyed the pace at which the book was produced: three months of writing and an extra month of revisions and production was the fastest project I ever worked on. Progressing at such speed, without sacrificing the quality of the end product, was extremely rewarding. Q. Why, in your opinion, is RabbitMQ exciting to discover, read, and write about? RabbitMQ is an open source project with a very active community: I'm convinced that open source can use any coverage it can receive, so writing a book about it was a way for me to pay back a little for the great piece of software I have been using for free. Moreover, though there were already many excellent books written about it, none had the brevity and mix of high- and low-level concerns I was envisioning for my book. Q. What is different about RabbitMQ from other open source message queuing software? The richness and interoperability of the AMQP protocol is an important factor for RabbitMQ's success. Another important factor is the solid engineering and sound design decisions that have been made by RabbitMQ's creators. The fact that it's built on Erlang brings some extra guarantees in terms of stability. Finally, the RabbitMQ team is excellent at offering powerful and complex features in an easy package: this is far from the norm in our industry. Q. What do you see on the horizon for RabbitMQ and message queuing, as a whole? RabbitMQ's popularity will keep rising, especially via cloud offerings that relieve users from the tedium of maintaining their own servers. In general, the usage of message queuing is poised to increase as developers become more and more aware of the principles at play when building scalable applications. The recent Reactive Manifesto (http://www.reactivemanifesto.org/), which somewhat rehashes and refreshes old principles of software design, emphasizes the necessity to decouple application components: message queuing is one of the main ways to achieve this. Q. Any tips for new authors? Writing a book is a fractal process where you proceed in several passes, with a higher level of details each time. The following approach has worked very well for me so it may work for others too: Start with the table of contents (TOC): write down the main ideas for each chapter and pay attention to the overall narrative, making sure that ideas develop progressively and logically When you write a chapter, start by copy/pasting the ideas from the TOC and flesh them out a little: don't write sentences yet but instead drop in notes and ideas for figures Prepare the code samples and capture screenshots of executing applications at that time Now you're ready for the last pass: finalize the chapter by writing the complete text and creating any extra diagrams needed Here are a few extra tips: Do not write in order: write paragraphs and even chapters in the way you feel most inspired to. This can relieve you from author's block. The first chapter is the hardest to write: get ready to come back to it several times. Find some music that helps you write: sometimes when you’re be tired and have a hard time getting started, music can get you back on track.

Jeremy, or Jay, is a Linux Administrator with over 12 years of experience and nine certifications. As a technologist, Jay enjoys all aspects of technology, and when not buried under a plethora of computer books, he enjoys music, photography, and writing. As well as tech books, Jay is also a published fiction author, having written his very own Sci-Fi novel, Escape to Planet 55. Jay's passion for open source software and its long term adoption led him to write Linux Mint Essentials for Packt. We asked Jay to discuss his experience of authoring a technology book with Packt and his insights on the future for open source technology. You can see his answers below and find more information about his books at LINKS. What initially drew you to write your book for Packt Publishing? When Packt approached me to write a book for Linux Mint, I was absolutely thrilled. I have always thought about how cool it would be to write a computer book, but never thought to actually attempt to do it. The thought of having a published book excited me. In addition, I also like very much how Packt donates proceeds back to the project being written about, so it felt good that I would help the Mint community in addition. When you began writing what were your main aims? What did you want your book to teach your readers? We've all been beginners at one point or another. For me, I started using Linux around 2002 when it was very difficult to get used to, and I didn't have much in the way of guidance or insight on how to use it. I stuck with it, and eventually became good at it. For my book, I wanted to make the process of getting accustomed to Linux as easy as possible, and for it to be the reference book I could have used at the time when I started. What did you enjoy most about the writing process and what was most rewarding about the experience of writing? The entire process was very rewarding, and fun. The experience I liked the most about writing was the fact that I was empowered to do it. For me, I like to teach others so I think the most rewarding part for me was the prospect of guiding others to enjoy using Linux as much as I have. If my book has this impact on others, then that will be the most rewarding part. What parts of the writing process were the most challenging and how did you overcome these challenges? The most challenging part of writing about open source software is how frequently it changes. During the writing process, two versions of Mint were released. This required going back to previous chapters and correcting things that were no longer true or had changed in some way. This was overcome by the rewrite phase of the project, where I had a chance to go back through the steps, provide new screenshots, and ensure the content was still compatible. Why, in your opinion, is Linux, or open source software, exciting to discover, read, and write about? Open source software, especially Linux, is extremely fun to learn and write about. I spend hours each day reading blogs, articles, and books, trying to keep up to date. Never does it feel tiring or laborious in any way. During my day job, I manage primarily Linux servers and workstations. When I get home, I read about open source and what's happening. While commuting, I listen to Linux-related podcasts (such as the Linux Action Show, Linux Unplugged, and so on) to keep current on upcoming trends. As I learn, I watch as my workflow changes and improves. While I initially found Linux difficult to learn back in 2002, nowadays I can't imagine my life without it! Why do you think interest in Linux, specifically Mint, is on the rise? I personally feel that Canonical (the makers of Ubuntu) are severely out of touch with what their users, or any user in general, wants and expects from a computing environment. I'm not saying that Ubuntu is a bad distribution, I actually like it. But the fact of the matter is that users have expectations of their computing environment. If these expectations are not meant (regardless of whether the expectations are logical or not) adoption will suffer. Linux Mint takes Ubuntu's base, improves on its weaknesses, and makes it much more convenient for the beginner user. In addition, Mint is scalable – it's perfect for a beginner, and is still very useful for when that beginner becomes an expert. People that care about how the internals of the distribution work will seek to learn about it, while those that don't care just want something that works. Whether you're a general user that just wants to check your Facebook account, or a developer writing the next big application – Mint fits just about every use case. What do you see on the horizon for Linux Mint, and Linux in general? In the short term, I think we'll continue to see Mint grow and expend its user base. Desktop environments, such as Cinnamon and MATE (featured heavily in Mint) will see quite a bit of expansion in the coming years, due to renewed developer focus. In the long term, I can see Mint splitting off into its own complete distribution, away from Ubuntu as its base. While there are no signs of that now, I can see Mint outgrowing its current base and moving off on its own in five years or so. Any tips/stories to share for aspiring/new technical authors? I think the best piece of advice is “yes, you can!” For me, I was very worried about whether or not I would even be good at this. When I sent my first chapter over, I thought the reaction would be that I was horrible and I would be excused from writing. I was really surprised to find out that Packt really liked what I was doing – even assuming that I had done this before! You never know what you're capable of until you give it a shot. I'm glad I did! Even if you're not the best writer in the world (I know I'm not) this is a valuable experience and you'll harness your writing skills as you go. Packt was there for me to work through the process, and it was very rewarding. Another piece of advice I can give is “just start writing.” Don't spend time worrying about how to write, what to say, or any of those bad things that can lead to writers block. Open up a word processor, or even a text editor, and just start writing something. You can always go back and correct/revise your sentences. The trick is to get your brain working, even if you don't plan on using what you're writing at that very minute. Just keep your mind busy and the rest will follow. Another important tip, which may seem like common knowledge to some, is to use some sort of versioning backup for your directory which includes your book files. A simple periodic copy to a flash drive or removable media isn't going to cut it, you want something that not only backs up your files but also allows you to go back to previous versions of a document in case you blow away content you didn't mean to. Examples of this include Dropbox, or Crashplan, though I'd recommend SpiderOak a bit more for its focus on security, a higher feature set, and the ability to sync between machines. All three are multi-platform.

IntroductionIn the ever-evolving world of web development, the need for rapid prototyping and efficient application creation has never been more paramount. "Web App Development Made Simple with Streamlit" is a book born out of this necessity. Streamlit, a powerful and intuitive tool, has transformed the way developers approach web app development, making it accessible to both seasoned programmers and beginners alike. This article delves into the core technologies discussed in the book, explores adjacent topics, and shares my personal journey in writing it.In-Depth Analysis of Streamlit TechnologyStreamlit is an open-source Python library that simplifies the development of web applications. Unlike traditional frameworks such as Django or Flask, Streamlit focuses on ease of use and rapid development. It allows developers to build interactive, data-driven web applications with minimal code.Key Features of Streamlit1. Simplicity and Speed: Streamlit's API is designed to be straightforward. With just a few lines of code, you can create a fully functional web app. This speed is crucial for data scientists and analysts who need to visualize data quickly. 2. Interactive Widgets: Streamlit provides a variety of widgets (e.g., sliders, buttons, and text inputs) that make it easy to create interactive applications. These widgets are essential for building user-friendly interfaces. 3. Real-Time Updates: Streamlit apps update in real-time as you modify the code, providing immediate feedback. This feature is particularly useful during the development phase, allowing for rapid iteration. 4. Integration with Python Ecosystem: Streamlit seamlessly integrates with popular Python libraries such as NumPy, Pandas, and Matplotlib. This compatibility ensures that developers can leverage existing tools and libraries.Solving Real-World ProblemsStreamlit addresses several challenges in web app development: Rapid Prototyping: Traditional web development frameworks often require significant setup and boilerplate code. Streamlit eliminates this overhead, enabling developers to focus on the core functionality of their applications. Data Visualization: Streamlit excels in creating data-driven applications. Its integration with visualization libraries allows developers to build insightful dashboards and reports with ease. - Accessibility: Streamlit's simplicity makes web development accessible to non-experts. Data scientists and analysts, who may not have extensive web development experience, can create powerful applications without needing to learn complex frameworks.Adjacent Topics: Best Practices and Emerging TrendsWhile Streamlit is a powerful tool, understanding best practices and emerging trends in web development is crucial for maximizing its potential.Best Practices1. Modular Code: Break your application into smaller, reusable components. This modular approach makes the codebase more manageable and easier to maintain. 2. Version Control: Use version control systems like Git to track changes and collaborate with others. Streamlit's simplicity should not deter you from following industry standard practices. 3. Testing: Implement automated testing to ensure your application behaves as expected. Libraries like pytest can be integrated with Streamlit applications for this purpose.Emerging Trends1. Low-Code/No-Code Platforms: Streamlit is part of a broader trend towards low-code and no-code platforms, which aim to democratize software development. These platforms enable users to build applications without extensive programming knowledge. 2. AI and Machine Learning Integration: Streamlit's ease of use makes it an ideal platform for deploying machine learning models. Expect to see more integration with AI tools and libraries in the near future. 3. Collaborative Development: Tools that facilitate collaboration among developers, data scientists, and domain experts are gaining traction.Streamlit's simplicity and real-time updates make it a natural fit for collaborative projects.My Writing JourneyWriting "Web App Development Made Simple with Streamlit" was a journey filled with both challenges and triumphs. The idea for the book stemmed from my own experiences as a developer struggling with the complexities of traditional web development frameworks. Iwanted to create a resource that would simplify the process and empower others to build their applications.Challenges 1. Staying Updated: The world of technology is constantly evolving. Keeping up with the latest updates and features in Streamlit was a significant challenge. I spent countless hours researching and experimenting to ensure the book was up-to-date. 2. Balancing Depth and Accessibility: Striking the right balance between providing in-depth technical information and making the content accessible to beginners was a delicate task. I aimed to cater to a broad audience without overwhelming them with jargon. 3. Time Management: Writing a book while juggling professional and personal commitments required meticulous time management. Setting realistic deadlines and sticking to a writing schedule was crucial.Overcoming Challenges1. Community Support: The Streamlit community was an invaluable resource. Engaging with other developers, participating in forums, and attending webinars helped me stay informed and inspired. 2. Iterative Writing: I adopted an iterative writing process, revisiting and refining chapters based on feedback from beta readers. This approach ensured the content was both accurate and user-friendly. 3. Personal Passion: My passion for simplifying web development kept me motivated. The thought of helping others overcome the same challenges I faced was a powerful driving force.Problem-Solving with StreamlitThe primary problem I aimed to solve with Streamlit was the steep learning curve associated with traditional web development frameworks. Many developers, especially those transitioning from data science or academia, struggle with the intricacies of setting up servers, managing databases, and creating user interfaces.Significance of the ProblemBarrier to Entry: The complexity of traditional frameworks can deter aspiring developers from pursuing web development. Streamlit lowers this barrier, making it accessible to a wider audience. Inefficiency: Time spent on boilerplate code and setup can be better utilized for developing core functionalities. Streamlit's simplicity allows developers to focus on solving real-world problems.SolutionStreamlit provides a streamlined approach to web development. By abstracting away the complexities, it empowers developers to create applications quickly and efficiently. My book offers practical examples and step-by-step guides to help readers harness the full potential of Streamlit.Unique Insights and Experiences"Web App Development Made Simple with Streamlit" offers several unique insights, drawn from my personal experiences and expertise.Key Insights 1. Real-World Applications: The book features case studies and real-world examples that demonstrate how Streamlit can be used to solve practical problems. These examples provide readers with a clear understanding of the tool's capabilities. 2. Hands-On Approach: Each chapter includes hands-on exercises and projects, encouraging readers to apply what they've learned. This interactive approach ensures a deeper understanding of the material. 3. Community Contributions: The book highlights contributions from the Streamlit community, showcasing innovative applications and best practices. This collaborative spirit reflects the open-source nature of Streamlit.Personal ExperiencesLearning from Mistakes: My journey with Streamlit was not without its mistakes. I share these experiences in the book, offering readers valuable lessons and tips to avoid common pitfalls. Continuous Learning: Writing the book was a learning experience in itself. I discovered new features, explored advanced use cases, and deepened my understanding of web development.Conclusion"Web App Development Made Simple with Streamlit" is more than just a technical guide; it's a testament to the power of simplicity and accessibility in web development. Streamlit has revolutionized the way we approach application development, making it possible for anyone to create powerful, interactive web apps. Through this article and my book, I hope to inspire and empower others to embark on their own development journeys, armed with the knowledge and tools to succeed.Author BioRosario Moscato has a master's degree in electronic engineering, a second level master in internet software design and a first level master's in science and faith. In about 25 years of experience, he has worked on innovative technology development in Europe and Asia. Recently, his interests have been focused exclusively on AI, pursuing the goal of making every business extremely competitive and analyzing the ethical implications deriving from the new scenarios that these disciplines open. Rosario has authored two books, and he is a speaker at international research centres and conferences as well as a trainer and technical/scientific consultant. Currently, he is working as CTO with one of the oldest AI companies in Italy.

School of Code is a company on a mission to help more people benefit from technology. And Packt are delighted to support the initiative. Empowering people to master code and feel confident developing software is central to Packt's mission, so it is fantastic to partner with an organization doing just that in the West Midlands.  We were lucky enough to speak to Nazia Choudrey who has been involved with the School of Code bootcamp. She told us about her experience with the initiative, and her perspective on what tech is like today for women and people of color.  Packt: Hi Nazia! Tell us a little about yourself. Nazia Choudrey: I am a person who is trying to re-invent myself. Just over a year ago I walked away from my so-called life to start afresh. I am not a person who looks back or has regrets, because everything that has happened is what has made me the person I am today – a person who has no fears and who is willing to try new things.  What were you doing before you enrolled for School of Code and what made you want to sign up? I was a probation court officer for 8 years, where I dealt with the day to day running of things especially focused on breaches. Even though I was on a computer every day I knew very little about how they worked, and more importantly, why they crashed on me. I guess that pursuit is what lead me to school of Code. What are you looking to gain from studying at School of Code in the long run? I’m looking to make computers my friend and, to be honest, to make a career in coding. With these new-found skills, I’ll hopefully be able to help others, develop products and software to help others gain a good standard of living which we are all entitled to. Do you think there is a diversity issue in the tech sector? Has it affected you in any way? I do think there is a diversity issue, as when most think about tech, they think of a white male person, an introvert, not a female from a minority group. Even I had that impression and therefore thought that this sector was not for me, but then I love to challenge sterotypes. I thought why not, let’s give it a try as I have nothing to lose. Why do you think making technology accessible to all is important? This is very important as, technology is taking over the world. Within the next 10-15 years all the jobs will be in this sector. Therefore, we all need to be ready to accept tech beyond our everyday lives. What do you think the future looks like for people working in the tech industry? Will larger companies strive to diversify their workforce, and, why should they? I think there is a bright future for people in the tech industry, I also think that all jobs will have some sort of tech that is needed. So therefore, this will need to be a diverse place where people from all sectors of life can come together. If large companies don't strive to be diverse they will be losing out on a large resource of social skills, of soft skills that diverse people can bring to the work place, putting them in positions to be global companies. School of Code encourages the honing of soft skills through networking, team work and project management. Do you think these skills are vital for the future of the Tech industry and attracting a new generation, shaking off the stereotype that all coders are solitary beings? Why? SoC helps and encourages people to work as a team, without the rat race to the finish. Find out more about School of Code  Download some of the books the Bootcampers found useful during the course: Thinking in HTML Thinking in CSS Thinking in JS series  MEAN Web Development React and React Native Responsive Web Design

This article is an excerpt from the book, "RAG-Driven Generative AI", by Denis Rothman. Explore the transformative potential of RAG-driven LLMs, computer vision, and generative AI with this comprehensive guide, from basics to building a complex RAG pipeline.IntroductionOn a bright Monday morning, Dakota sits down to get to work and is called by the CEO of their software company, who looks quite worried. An important fire department needs a conversational AI agent to train hundreds of rookie firefighters nationwide on drone technology. The CEO looks dismayed because the data provided is spread over many websites around the country. Worse, the management of the fire department is coming over at 2 PM to see a demonstration to decide whether to work with Dakata’s company or not. Dakota is smiling. The CEO is puzzled.  Dakota explains that the AI team can put a prototype together in a few hours and be more than ready by 2 PM and get to work. The strategy is to divide the AI team into three sub-teams that will work in parallel on three pipelines based on the reference Deep Lake, LlamaIndex and OpenAI RAG program* they had tested and approved a few weeks back.  Pipeline 1: Collecting and preparing the documents provided by the fire department for this Proof of Concept(POC). Pipeline 2: Creating and populating a Deep Lake vector store with the first batch of documents while the Pipeline 1 team continues to retrieve and prepare the documents. Pipeline 3: Indexed-based RAG with LlamaIndex’s integrated OpenAI LLM performed on the first batch of vectorized documents. The team gets to work at around 9:30 AM after devising their strategy. The Pipeline 1 team begins by fetching and cleaning a batch of documents. They run Python functions to remove punctuation except for periods and noisy references within the content. Leveraging the automated functions they already have through the educational program, the result is satisfactory.  By 10 AM, the Pipeline 2 team sees the first batch of documents appear on their file server. They run the code they got from the RAG program* to create a Deep Lake vector store and seamlessly populate it with an OpenAI embedding model, as shown in the following excerpt: from llama_index.core import StorageContext vector_store_path = "hub://denis76/drone_v2" dataset_path = "hub://denis76/drone_v2" # overwrite=True will overwrite dataset, False will append it vector_store = DeepLakeVectorStore(dataset_path=dataset_path, overwrite=True)  Note that the path of the dataset points to the online Deep Lake vector store. The fact that the vector store is serverless is a huge advantage because there is no need to manage its size, storage process and just begin to populate it in a few seconds! Also, to process the first batch of documents, overwrite=True, will force the system to write the initial data. Then, starting the second batch,  the Pipeline 2 team can run overwrite=False, to append the following documents. Finally,  LlamaIndex automatically creates a vector store index: storage_context = StorageContext.from_defaults(vector_store=vector_store) # Create an index over the documents index = VectorStoreIndex.from_documents(documents, storage_context=storage_context) By 10:30 AM, the Pipeline 3 team can visualize the vectorized(embedded) dataset in their Deep Lake vector store. They create a LlamaIndex query engine on the dataset: from llama_index.core import VectorStoreIndex vector_store_index = VectorStoreIndex.from_documents(documents) … vector_query_engine = vector_store_index.as_query_engine(similarity_top_k=k, temperature=temp, num_output=mt) Note that the OpenAI Large Language Model is seamlessly integrated into LlamaIndex with the following parameters: k, in this case, k=3, specifies the number of documents to retrieve from the vector store. The retrieval is based on the similarity of embedded user inputs and embedded vectors within the dataset. temp, in this case temp=0.1, determines the randomness of the output. A low value such as 0.1 forces the similarity search to be precise. A higher value would allow for more diverse responses, which we do not want for this technological conversational agent. mt, in this case, mt=1024, determines the maximum number of tokens in the output. A cosine similarity function was added to make sure that the outputs matched the sample user inputs: from sentence_transformers import SentenceTransformer model = SentenceTransformer('all-MiniLM-L6-v2') def calculate_cosine_similarity_with_embeddings(text1, text2):     embeddings1 = model.encode(text1)     embeddings2 = model.encode(text2)     similarity = cosine_similarity([embeddings1], [embeddings2])     return similarity[0][0] By 11:00 AM, all three pipeline teams are warmed up and ready to go full throttle! While the Pipeline 2 team was creating the vector store and populating it with the first batch of documents, the Pipeline 1 team prepared the next several batches. At 11:00 AM, Dakota gave the green light to run all three pipelines simultaneously. Within a few minutes, the whole RAG-driven generative AI system was humming like a beehive! By 1:00 PM, Dakota and the three pipeline teams were working on a PowerPoint slideshow with a copilot. Within a few minutes, it was automatically generated based on their scenario. At 1:30 PM, they had time to grab a quick lunch. At 2:00 pm, the fire department management, Dakota’s team, and the CEO of their software company were in the meeting room.  Dakota’s team ran the PowerPoint slide show and began the demonstration with a simple input:  user_input="Explain how drones employ real-time image processing and machine learning algorithms to accurately detect events in various environmental conditions." The response displayed was satisfactory: Drones utilize real-time image processing and machine learning algorithms to accurately detect events in various environmental conditions by analyzing data captured by their sensors and cameras. This technology allows drones to process visual information quickly and efficiently, enabling them to identify specific objects, patterns, or changes in the environment in real-time. By employing these advanced algorithms, drones can effectively monitor and respond to different situations, such as wildfires, wildlife surveys, disaster relief efforts, and agricultural monitoring with precision and accuracy. Dakota’s team then showed that the program could track and display the original documents the response was based on. At one point, the fire department’s top manager, Taylor, exclaimed, “Wow, this is impressive! It’s exactly what we were looking for! " Of course, Dakato’s CEO began discussing the number of users, cost, and timelines with Taylor. In the meantime, Dakota and the rest of the fire department’s team went out to drink some coffee and get to know each other. Fire departments intervene at short notice efficiently for emergencies. So can expert-level AI teams! https://github.com/Denis2054/RAG-Driven-Generative-AI/blob/main/Chapter03/Deep_Lake_LlamaIndex_OpenAI_RAG.ipynb ConclusionIn facing a high-stakes, time-sensitive challenge, Dakota and their AI team demonstrated the power and efficiency of RAG-driven generative AI. By leveraging a structured, multi-pipeline approach with tools like Deep Lake, LlamaIndex, and OpenAI’s advanced models, the team was able to integrate scattered data sources quickly and effectively, delivering a sophisticated, real-time conversational AI prototype tailored for firefighter training on drone technology. Their success showcases how expert planning, resourceful use of AI tools, and teamwork can transform a complex project into a streamlined solution that meets client needs. This case underscores the potential of generative AI to create responsive, practical solutions for critical industries, setting a new standard for rapid, high-quality AI deployment in real-world applications.Author Bio Denis Rothman graduated from Sorbonne University and Paris-Diderot University, and as a student, he wrote and registered a patent for one of the earliest word2vector embeddings and word piece tokenization solutions. He started a company focused on deploying AI and went on to author one of the first AI cognitive NLP chatbots, applied as a language teaching tool for Mo�t et Chandon (part of LVMH) and more. Denis rapidly became an expert in explainable AI, incorporating interpretable, acceptance-based explanation data and interfaces into solutions implemented for major corporate projects in the aerospace, apparel, and supply chain sectors. His core belief is that you only really know something once you have taught somebody how to do it.