Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Free Learning
Arrow right icon

How-To Tutorials - Cloud Computing

121 Articles
article-image-the-10-best-cloud-and-infrastructure-conferences-happening-in-2019
Sugandha Lahoti
23 Jan 2019
11 min read
Save for later

The 10 best cloud and infrastructure conferences happening in 2019

Sugandha Lahoti
23 Jan 2019
11 min read
The latest Gartner report suggests that the cloud market is going to grow an astonishing 17.3% ($206 billion) in 2019, up from $175.8 billion in 2018. By 2022, the report claims, 90% of organizations will be using cloud services. But the cloud isn’t one thing, and 2019 is likely to bring the diversity of solutions, from hybrid to multi-cloud, to serverless, to the fore. With such a mix of opportunities and emerging trends, it’s going to be essential to keep a close eye on key cloud computing and software infrastructure conferences throughout the year. These are the events where we’ll hear the most important announcements, and they’ll probably also be the place where the most important conversations happen too. But with so many cloud computing conferences dotted throughout the year, it’s hard to know where to focus your attention. For that very reason, we’ve put together a list of some of the best cloud computing conferences taking place in 2019. #1 Google Cloud Next When and where is Google Cloud Next 2019 happening? April 9-11 at the Moscone Center in San Francisco. What is it? This is Google’s annual global conference focusing on the company’s cloud services and products, namely Google Cloud Platform. At previous events, Google has announced enterprise products such as G Suite and Developer Tools. The three-day conference features demonstrations, keynotes, announcements, conversations, and boot camps. What’s happening at Google Cloud Next 2019? This year Google Cloud Next has more than 450 sessions scheduled. You can also meet directly with Google experts in artificial intelligence and machine learning, security, and software infrastructure. Themes covered this year include application development, architecture, collaboration, and productivity, compute, cost management, DevOps and SRE, hybrid cloud, and serverless. The conference may also serve as a debut platform for new Google Cloud CEO Thomas Kurian. Who’s it for? The event is a not-to-miss event for IT professionals and engineers, but it will also likely attract entrepreneurs. For those of us who won’t attend, Google Cloud Next will certainly be one of the most important conferences to follow. Early bird registration begins from March 1 for $999. #2 OpenStack Infrastructure Summit When and where is OpenStack Infrastructure Summit 2019 happening? April 29 - May 1 in Denver. What is it? The OpenStack Infrastructure Summit, previously the OpenStack Summit, is focused on open infrastructure integration and has evolved over the years to cover more than 30 different open source projects.  The event is structured around use cases, training, and related open source projects. The summit also conducts Project Teams Gathering, just after the main conference (this year May 2-4). PTG provides meeting facilities, allowing various technical teams contributing to OSF (Open Science Framework) projects to meet in person, exchange ideas and get work done in a productive setting. What’s happening at this year’s OpenStack Infrastructure Summit? This year the summit is expected to have almost 300 sessions and workshops on Container Infrastructure, CI/CD, Telecom + NFV, Public Cloud, Private & Hybrid Cloud, Security etc. The Summit is going to have members of open source communities like Airship, Ansible, Ceph, Docker, Kata Containers, Kubernetes, ONAP, OpenStack, Open vSwitch, OPNFV, StarlingX, Zuul among other topics. Who’s it for? This is an event for engineers working in operations and administration. If you’re interested in OpenStack and how the foundation fits into the modern cloud landscape there will certainly be something here for you. #3 DockerCon When and where is DockerCon 2019 happening? April 29 to May 2 at Moscone West, San Francisco. What is it? DockerCon is perhaps the container event of the year. The focus is on what’s happening across the Docker world, but it will offer plenty of opportunities to explore the ways Docker is interacting and evolving with a wider ecosystem of tools. What’s happening at DockerCon 2019? This three-day conference will feature networking opportunities and hands-on labs. It will also hold an exposition where innovators will showcase their latest products. It’s expected to have over 6,000 attendees with 5+ tracks and 100 sessions. You’ll also have the opportunity to become a Docker Certified Associate with an on-venue test. Who’s it for? The event is essential for anyone working in and around containers - so DevOps, SRE, administration and infrastructure engineers. Of course, with Docker finding its way into the toolsets of a variety of roles, it may be useful for people who want to understand how Docker might change the way they work in the future.  Pricing for DockerCon runs from around $1080 for early-bird reservations to $1350 for standard tickets. #4 Red Hat Summit When and where is Red Hat Summit 2019 happening? May 7–9 in Boston. What is it? Red Hat Summit is an open source technology event run by Red Hat. It covers a wide range of topics and issues, essentially providing a snapshot of where the open source world is at the moment and where it might be going. With open source shaping cloud and other related trends, it’s easy to see why the event could be important for anyone with an interest in cloud and infrastructure. What’s happening at Red Hat Summit 2019? The theme for this year is AND. The copy on the event’s website reads:  AND is about scaling your technology and culture in whatever size or direction you need, when you need to, with what you actually need―not a bunch of bulky add-ons. From the right foundation―an open foundation―AND adapts with you. It’s interoperable, adjustable, elastic. Think Linux AND Containers. Think public AND private cloud. Think Red Hat AND you. There’s clearly an interesting conceptual proposition at the center of this year’s event that hints at how Red Hat wants to get engineers and technology buyers to think about the tools they use and how they use them. Who’s it for? The event is big for any admin or engineer that works with open source technology - Linux in particular (so, quite a lot of people…). Given Red Hat was bought by IBM just a few months ago in 2018, this event will certainly be worth watching for anyone interested in the evolution of both companies as well as open source software more broadly. #5 KubeCon + CloudNativeCon Europe When and where is KubeCon + CloudNativeCon Europe 2019? May 20 to 23 at Fira Barcelona. What is it? KubeCon + CloudNativeCon is CCNF’s (Cloud Native Computing Foundation) flagship conference for open source and cloud-native communities. It features contributors from cloud-native applications and computing, containers, microservices, central orchestration processing, and related projects to further cloud-native education of technologies that support the cloud-native ecosystem. What’s happening at this year’s KubeCon? The conference will feature a range of events and sessions from industry experts, project leaders, as well as sponsors. The details of the conference still need development, but the focus will be on projects such as Kubernetes (obviously), Prometheus, Linkerd, and CoreDNS. Who’s it for? The conference is relevant to anyone with an interest in software infrastructure. It’s likely to be instructive and insightful for those working in SRE, DevOps and administration, but because of Kubernetes importance in cloud native practices, there will be something here for many others in the technology industry. . The cost is unconfirmed, but it can be anywhere between $150 and $1,100. #6 IEEE International Conference on Cloud Computing When and where is the IEEE International Conference on Cloud Computing? July 8-13 in Milan. What is it? This is an IEEE conference solely dedicated to Cloud computing. IEEE Cloud is basically for research practitioners to exchange their findings on the latest cloud computing advances. It includes findings across all “as a service” categories, including network, infrastructure, platform, software, and function. What’s happening at the IEEE International Conference on Cloud Computing? IEEE cloud 2019 invites original research papers addressing all aspects of cloud computing technology, systems, applications, and business innovations. These are mostly based on technical topics including cloud as a service, cloud applications, cloud infrastructure, cloud computing architectures, cloud management, and operations. Shangguang Wang and Stephan Reiff-Marganiec have been appointed as congress workshops chairs. Featured keynote speakers for the 2019 World Congress on Services include Kathryn Guarini, VP at IBM Industry Research and Joseph Sifakis, the Emeritus Senior CNRS Researcher at Verimag. Who’s it for? The conference has a more academic bent than the others on this list. That means it’s particularly important for researchers in the field, but there will undoubtedly be lots here for industry practitioners that want to find new perspectives on the relationship between cloud computing and business. #7 VMworld When and where is VMWorld 2019? August 25 - 29 in San Francisco. What is it? VMworld is a virtualization and cloud computing conference, hosted by VMware. It is the largest virtualization-specific event. VMware CEO Pat Gelsinger and the executive team typically provide updates on the company’s various business strategies, including multi-cloud management, VMware Cloud for AWS, end-user productivity, security, mobile, and other efforts. What’s happening at VMworld 2019? The 5-day conference starts with general sessions on IT and business. It then goes deeper into breakout sessions, expert panels, and quick talks. It also holds various VMware Hands-on Labs and VMware Certification opportunities as well as one-on-one appointments with in-house experts. The expected attendee is over 21000+. Who’s it for? VMworld maybe doesn’t have the glitz and glamor of an event like DockerCon or KubeCon, but for administrators and technological decision makers that have an interest in VMware’s products and services. #8 Microsoft Ignite When and where is Microsoft Ignite 2019? November 4-8 at Orlando, Florida What is it? Ignite is Microsoft's flagship enterprise event for everything cloud, data, business intelligence, teamwork, and productivity. What’s happening at Microsoft Ignite 2019? Microsoft Ignite 2019 is expected to feature almost 700 + deep-dive sessions and 100 + expert-led and self-paced workshops. The full agenda will be posted sometime in Spring 2019. You can pre-register for Ignite 2019 here. Microsoft will also be touring many cities around the world to bring the Ignite experience to more people. Who’s it for? The event should have wide appeal, and will likely reflect Microsoft’s efforts to bring a range of tech professionals into the ecosystem. Whether you’re a developer, infrastructure engineer, or operations manager, Ignite is, at the very least, an event you should pay attention to. #9 Dreamforce When and where is Dreamforce 2019? November 19-22, in San Francisco. What is it? Dreamforce, hosted by Salesforce, is a truly huge conference, attended by more than 100,000 people.. Focusing on Salesforce and CRM, the event is an opportunity to learn from experts, share experiences and ideas, and to stay up to speed with the trends in the field, like automation and artificial intelligence. What’s happening at Dreamforce 2019? Dreamforce covers over 25 keynotes, a vast range of breakout sessions (almost 2700) and plenty of opportunities for networking. The conference is so extensive that it has its own app to help delegates manage their agenda and navigate venues. Who’s it for? Dreamforce is primarily about Salesforce - for that reason, it’s very much an event for customers and users. But given the size of the event, it also offers a great deal of insight on how businesses are using SaaS products and what they expect from them. This means there is plenty for those working in more technical or product roles to learn at the event.. #10 Amazon re:invent When and where is Amazon re:invent 2019? December 2-6 at The Venetian, Las Vegas, USA What is it? Amazon re:invent is hosted by AWS. If you’ve been living on mars in recent years, AWS is the market leader when it comes to cloud. The event, then, is AWS’ opportunity to set the agenda for the cloud landscape, announcing updates and new features, as well as an opportunity to discuss the future of the platform. What’s happening at Amazon re:invent 2019? Around 40,000 people typically attend Amazon’s top cloud event.  Amazon Web Services and its cloud-focused partners typically reveal product releases on several fronts. Some of these include enterprise security, Transit Virtual Private Cloud service, and general releases. This year, Amazon is also launching a related conference dedicated exclusively to cloud security called re:Inforce. The inaugural event will take place in Boston on June 25th and 26th, 2019 at the Boston Convention and Exhibition Center. Who’s it for? The conference attracts Amazon’s top customers, software distribution partners (ISVs) and public cloud MSPs. The event is essential for developers and engineers, administrators, architects, and decision makers. Given the importance of AWS in the broader technology ecosystem, this is an event that will be well worth tracking, wherever you are in the world. Did we miss an important cloud computing conference? Are you attending any of these this year? Let us know in the comments – we’d love to hear from you. Also, check this space for more detailed coverage of the conferences. Cloud computing trends in 2019 Key trends in software development in 2019: cloud native and the shrinking stack Key trends in software infrastructure in 2019: observability, chaos, and cloud complexity
Read more
  • 0
  • 0
  • 11342

article-image-cloud-computing-trends-in-2019
Guest Contributor
07 Jan 2019
8 min read
Save for later

Cloud computing trends in 2019

Guest Contributor
07 Jan 2019
8 min read
Cloud computing is a rapidly growing technology that many organizations are adopting to enable their digital transformation. As per the latest Gartner report, the cloud tech services market is projected to grow 17.3% ($206 billion) in 2019, up from $175.8 billion in 2018 and by 2022, 90% of organizations will be using cloud services. In today’s world, Cloud technology is a trending buzzword among business environments. It provides exciting new opportunities for businesses to compete on a global scale and is redefining the way we do business. It enables a user to store and share data like applications, files, and more to remote locations. These features have been realized by all business owners, from startup to well-established organizations, and they have already started using cloud computing. How Cloud technology helps businesses Reduced Cost One of the most obvious advantages small businesses can get by shifting to the cloud is saving money. It can provide small business with services at affordable and scalable prices. Virtualization expands the value of physical equipment, which means companies can achieve more with less. Therefore, an organization can see a significant decline in power consumption, rack space, IT requirements, and more. As a result, there is lower maintenance, installation, hardware, support & upgrade costs. For small businesses, particularly, those savings are essential. Enhanced Flexibility Cloud can access data and related files from any location and from any device at any time with an internet connection. As the working process is changing to flexible and remote working, it is essential to provide work-related data access to employees, even when they are not at a workplace. Cloud computing not only helps employees to work outside of the office premises but also allows employers to manage their business as and when required. Also, enhanced flexibility & mobility in cloud technology can lead to additional cost savings. For example, an employer can select to execute BYOD (bring your own device). Therefore, employees can bring and work on their own devices which they are comfortable in.. Secured Data Improved data security is another asset of cloud computing. With traditional data storage systems, the data can be easily stolen or damaged. There can also be more chances for serious cyber attacks like viruses, malware, and hacking. Human errors and power outages can also affect data security. However, if you use cloud computing, you will get the advantages of improved data security. In the cloud, the data is protected in various ways such as anti-virus, encryption methods, and many more. Additionally, to reduce the chance of data loss, the cloud services help you to remain in compliance with HIPAA, PCI, and other regulations. Effective Collaboration Effective collaboration is possible through the cloud which helps small businesses to track and oversee workflow and progress for effective results. There are many cloud collaboration tools available in the market such as Google Drive, Salesforce, Basecamp, Hive, etc. These tools allow users to create, edit, save and share documents for workplace collaboration. A user can also constrain the access of these materials. Greater Integration Cloud-based business solutions can create various simplified integration opportunities with numerous cloud-based providers. They can also get benefits of specialized services that integrate with back-office operations such as HR, accounting, and marketing. This type of integration makes business owners concentrate on the core areas of a business. Scalability One of the great aspects of cloud-based services is their scalability. Currently, a small business may require limited storage, mobility, and more. But in future, needs & requirements will increase significantly in parallel with the growth of the business.  Considering that growth does not always occur linearly, cloud-based solutions can accommodate all sudden and increased requirements of the organization. Cloud-based services have the flexibility to scale up or to scale down. This feature ensures that all your requirements are served according to your budget plans. Cloud Computing Trends in 2019 Hybrid & Multi-Cloud Solutions Hybrid Cloud will become the dominant business model in the future. For organizations, the public cloud cannot be a good fit for all type of solutions and shifting everything to the cloud can be a difficult task as they have certain requirements. The Hybrid Cloud model offers a transition solution that blends the current on-premises infrastructure with open cloud & private cloud services. Thus, organizations will be able to shift to the cloud technology at their own pace while being effective and flexible. Multi-Cloud is the next step in the cloud evolution. It enables users to control and run an application, workload, or data on any cloud (private, public and hybrid) based on their technical requirements. Thus, a company can have multiple public and private clouds or multiple hybrid clouds, all either connected together or not. We can expect multi-cloud strategies to dominate in the coming days. Backup and Disaster Recovery According to Spiceworks report,  15% of the cloud budget is allocated to Backup and Disaster Recovery (DR) solutions, which is the highest budget allocation, followed by email hosting and productivity tools. This huge percentage impacts the shared responsibility model that public cloud providers operate on. Public cloud providers, like as AWS (Amazon Web Services ), Microsoft Azure, Google Cloud are responsible for the availability of Backup and DR solutions and security of the infrastructure, while the users are in charge for their data protection and compliance. Serverless Computing Serverless Computing is gaining more popularity and will continue to do so in 2019. It is a procedure utilized by Cloud users, who request a container PaaS (Platform as a Service), and Cloud supplier charges for the PaaS as required. The customer does not need to buy or rent services before and doesn't need to configure them. The Cloud is responsible for providing the platform, it’s configuration, and a wide range of helpful tools for designing applications, and working with data. Data Containers The process of Data Container usage will become easier in 2019. Containers are more popular for transferring data, they store and organize virtual objects, and resolve the issues of having software run reliably while transferring the data from one system to another. However, there are some confinements. While containers are used to transport, they can only be used with servers having compatible operating system “kernels.” Artificial Intelligence Platforms The utilization of AI to process Big Data is one of the more important upgrades in collecting business intelligence data and giving a superior comprehension of how business functions. AI platform supports a faster, more effective, and more efficient approach to work together with data scientists and other team members. It can help to reduce costs in a variety of ways, such as making simple tasks automated, preventing the duplication of effort, and taking over some expensive labor tasks, such as copying or extraction of data. Edge computing Edge computing is a systematic approach to execute data processing at the edge of the network to streamline cloud computing. It is a result of ever increased use of IoT devices. Edge is essential to run real-time services as it streamlines the flow of traffic from IoT devices and provides real-time data analytics and analysis. Hence, it is also on the rise in 2019. Service mesh Service mesh is a dedicated system layer to enhance service to service communication across microservices applications. It's a new and emerging class of service management for the inter-microservice communication complexity and provides observability and tracing in a seamless way. As containers become more prevalent for cloud-based application development, the requirement for service mesh is increasing significantly. Service meshes can help oversee traffic through service discovery, load balancing, routing, and observability. Service meshes attempt to diminish the complexity of containers and improve network functionality. Cloud Security As we see the rise in technology, security is obviously another serious consideration. With the introduction of the GDPR (General Data Protection Regulation) security concerns have risen much higher and are the essential thing to look after. Many businesses are shifting to cloud computing without any serious consideration of its security compliance protocols. Therefore, GDPR will be an important thing in 2019 and the organization must ensure that their data practices are both safe and compliant. Conclusion As we discussed above, cloud technology is capable of providing better data storage, data security, collaboration, and it also changes the workflow to help small business owners to take better decisions. Finally, cloud connectivity is all about convenience, and streamlining workflow to help any business become more flexible, efficient, productive, and successful. If you want to set your business up for success, this might be the time to transition to cloud-based services. Author Bio Amarendra Babu L loves pursuing excellence through writing and has a passion for technology. He is presently working as a content contributor for Mindmajix.com and Tekslate.com. He is a tech-geek and love to explore new opportunities. His work has been published on various sites related to Big Data, Business Analytics & Intelligence, Blockchain, Cloud Computing, Data Science, AI & ML, Project Management, and more. You can reach him at amarendrabl18@gmail.com. He is also available on Linkedin. 8 programming languages to learn in 2019 18 people in tech every programmer and software engineer need to follow in 2019 We discuss the key trends for web and app developers in 2019 [Podcast]
Read more
  • 0
  • 0
  • 15524

article-image-key-trends-in-software-infrastructure-in-2019
Richard Gall
17 Dec 2018
10 min read
Save for later

Key trends in software infrastructure in 2019: observability, chaos, and cloud complexity

Richard Gall
17 Dec 2018
10 min read
Software infrastructure has, over the last decade or so, become a key concern for developers of all stripes. Long gone are narrowly defined job roles; thanks to DevOps, accountability for how code is now shared between teams on both development and deployment sides. For anyone that’s ever been involved in the messy frustration of internal code wars, this has been a welcome change. But as developers who have traditionally sat higher up the software stack dive deeper into the mechanics of deploying and maintaining software, for those of us working in system administration, DevOps, SRE, and security (the list is endless, apologies if I’ve forgotten you), the rise of distributed systems only brings further challenges. Increased complexity not only opens up new points of failure and potential vulnerability, at a really basic level it makes understanding what’s actually going on difficult. And, essentially, this is what it will mean to work in software delivery and maintenance in 2019. Understanding what’s happening, minimizing downtime, taking steps to mitigate security threats - it’s a cliche, but finding strategies to become more responsive rather than reactive will be vital. Indeed, many responses to these kind of questions have emerged this year. Chaos engineering and observability, for example, have both been gaining traction within the SRE world, and are slowly beginning to make an impact beyond that particular job role. But let’s take a deeper look at what is really going to matter in the world of software infrastructure and architecture in 2019. Observability and the rise of the service mesh Before we decide what to actually do, it’s essential to know what’s actually going on. That seems obvious, but with increasing architectural complexity, that’s getting harder. Observability is a term that’s being widely thrown around as a response to this - but it has been met with some cynicism. For some developers, observability is just a sexed up way of talking about good old fashioned monitoring. But although the two concepts have a lot in common, observability is more of an approach, a design pattern maybe, rather than a specific activity. This post from The New Stack explains the difference between monitoring and observability incredibly well. Observability is “a measure of how well internal states of a system can be inferred from knowledge of its external outputs.” which means observability is more a property of a system, rather than an activity. There are a range of tools available to help you move towards better observability. Application management and logging tools like Splunk, Datadog, New Relic and Honeycomb can all be put to good use and are a good first step towards developing a more observable system. Want to learn how to put monitoring tools to work? Check out some of these titles: AWS Application Architecture and Management [Video]     Hands on Microservices Monitoring and Testing       Software Architecture with Spring 5.0      As well as those tools, if you’re working with containers, Kubernetes has some really useful features that can help you more effectively monitor your container deployments. In May, Google announced StackDriver Kubernetes Monitoring, which has seen much popularity across the community. Master monitoring with Kubernetes. Explore these titles: Google Cloud Platform Administration     Mastering Kubernetes      Kubernetes in 7 Days [Video]        But there’s something else emerging alongside observability which only appears to confirm it’s importance: that thing is the notion of a service mesh. The service mesh is essentially a tool that allows you to monitor all the various facets of your software infrastructure helping you to manage everything from performance to security to reliability. There are a number of different options out there when it comes to service meshes - Istio, Linkerd, Conduit and Tetrate being the 4 definitive tools out there at the moment. Learn more about service meshes inside these titles: Microservices Development Cookbook     The Ultimate Openshift Bootcamp [Video]     Cloud Native Application Development with Java EE [Video]       Why is observability important? Observability is important because it sets the foundations for many aspects of software management and design in various domains. Whether you’re an SRE or security engineer, having visibility on the way in which your software is working will be essential in 2019. Chaos engineering Observability lays the groundwork for many interesting new developments, chaos engineering being one of them. Based on the principle that modern, distributed software is inherently unreliable, chaos engineering ‘stress tests’ software systems. The word ‘chaos’ is a bit of a misnomer. All testing and experimentation on your software should follow a rigorous and almost scientific structure. Using something called chaos experiments - adding something unexpected into your system, or pulling a piece of it out like a game of Jenga - chaos engineering helps you to better understand the way it will act in various situations. In turn, this allows you to make the necessary changes that can help ensure resiliency. Chaos engineering is particularly important today simply because so many people, indeed, so many things, depend on software to actually work. From an eCommerce site to a self driving car, if something isn’t working properly there could be terrible consequences. It’s not hard to see how chaos engineering fits alongside something like observability. To a certain extent, it’s really another way of achieving observability. By running chaos experiments, you can draw out issues that may not be visible in usual scenarios. However, the caveat is that chaos engineering isn’t an easy thing to do. It requires a lot of confidence and engineering intelligence. Running experiments shouldn’t be done carelessly - in many ways, the word ‘chaos’ is a bit of a misnomer. All testing and experimentation on your software should follow a rigorous and almost scientific structure. While chaos engineering isn’t straightforward, there are tools and platforms available to make it more manageable. Gremlin is perhaps the best example, offering what they describe as ‘resiliency-as-a-service’. But if you’re not ready to go in for a fully fledged platform, it’s worth looking at open source tools like Chaos Monkey and ChaosToolkit. Want to learn how to put the principles of chaos engineering into practice? Check out this title: Microservice Patterns and Best Practices       Learn the principles behind resiliency with these SRE titles: Real-World SRE       Practical Site Reliability Engineering       Better integrated security and code testing Both chaos engineering and observability point towards more testing. And this shouldn’t be surprising: testing is to be expected in a world where people are accountable for unpredictable systems. But what’s particularly important is how testing is integrated. Whether it’s for security or simply performance, we’re gradually moving towards a world where testing is part of the build and deploy process, not completely isolated from it. There are a diverse range of tools that all hint at this move. Archery, for example, is a tool designed for both developers and security testers to better identify and assess security vulnerabilities at various stages of the development lifecycle. With a useful dashboard, it neatly ties into the wider trend of observability. ArchUnit (sounds similar but completely unrelated) is a Java testing library that allows you to test a variety of different architectural components. Similarly on the testing front, headless browsers continue to dominate. We’ve seen some of the major browsers bringing out headless browsers, which will no doubt delight many developers. Headless browsers allow developers to run front end tests on their code as if it were live and running in the browser. If this sounds a lot like PhantomJS, that’s because it is actually quite a bit like PhantomJS. However, headless browsers do make the testing process much faster. Smarter software purchasing and the move to hybrid cloud The key trends we’ve seen in software architecture are about better understanding your software. But this level of insight and understanding doesn’t matter if there’s no alignment between key decision makers and purchasers. Whatever cloud architecture you have, strong leadership and stakeholder management are essential. This can manifest itself in various ways. Essentially, it’s a symptom of decision makers being disconnected from engineers buried deep in their software. This is by no means a new problem, cloud coming to define just about every aspect of software, it’s now much easier for confusion to take hold. The best thing about cloud is also the worst thing - the huge scope of opportunities it opens up. It makes decision making a minefield - which provider should we use? What parts of it do we need? What’s going to be most cost effective? Of course, with hybrid cloud, there's a clear way of meeting those issues. But it's by no means a silver bullet. Whatever cloud architecture you have, strong leadership and stakeholder management are essential. This is something that ThoughtWorks references in its most recent edition of Radar (November 2018). Identifying two trends they call ‘bounded buy’ and ‘risk commensurate vendor strategy’ ThoughtWorks highlights how organizations can find their SaaS of choice shaping their strategy in its own image (bounded buy) or look to outsource business critical applications, functions or services. T ThoughtWorks explains: “This trade-off has become apparent as the major cloud providers have expanded their range of service offerings. For example, using AWS Secret Management Service can speed up initial development and has the benefit of ecosystem integration, but it will also add more inertia if you ever need to migrate to a different cloud provider than it would if you had implemented, for example, Vault”. Relatedly, ThoughtWorks also identifies a problem with how organizations manage cost. In the report they discuss what they call ‘run cost as architecture fitness function’ which is really an elaborate way of saying - make sure you look at how much things cost. So, for example, don’t use serverless blindly. While it might look like a cheap option for smaller projects, your costs could quickly spiral and leave you spending more than you would if you ran it on a typical cloud server. Get to grips with hybrid cloud: Hybrid Cloud for Architects       Building Hybrid Clouds with Azure Stack     Become an effective software and solutions architect in 2019: AWS Certified Solutions Architect - Associate Guide     Architecting Cloud Computing Solutions     Hands-On Cloud Solutions with Azure       Software complexity needs are best communicated in a simple language: money In practice, this takes us all the way back to the beginning - it’s simply the financial underbelly of observability. Performance, visibility, resilience - these matter because they directly impact the bottom line. That might sound obvious, but if you’re trying to make the case, say, for implementing chaos engineering, or using a any other particular facet of a SaaS offering, communicating to other stakeholders in financial terms can give you buy-in and help to guarantee alignment. If 2019 should be about anything, it’s getting closer to this fantasy of alignment. In the end, it will keep everyone happy - engineers and businesses
Read more
  • 0
  • 0
  • 6068
Banner background image

article-image-the-future-of-cloud-lies-in-revisiting-the-designs-and-limitations-of-todays-notion-of-serverless-computing-say-uc-berkeley-researchers
Savia Lobo
17 Dec 2018
5 min read
Save for later

The Future of Cloud lies in revisiting the designs and limitations of today’s notion of ‘serverless computing’, say UC Berkeley researchers

Savia Lobo
17 Dec 2018
5 min read
Last week, researchers at the UC Berkeley released a research paper titled ‘Serverless Computing: One Step Forward, Two Steps Back’, which highlights some pitfalls in the current serverless architectures. Researchers have also explored the challenges that should be addressed to utilize the complete potential that the cloud can offer to innovative developers. Cloud isn’t being used to the fullest The researchers have termed cloud as “the biggest assemblage of data capacity and distributed computing power ever available to the general public, managed as a service”. The cloud today is being used as an outsourcing platform for standard enterprise data services. In order to leverage the actual potential of the cloud to the fullest, creative developers need programming frameworks. The majority of cloud services are simply multi-tenant, easier-to-administer clones of legacy enterprise data services such as object storage, databases, queueing systems, and web/app servers. Off late, the buzz for serverless computing--a platform in the cloud where developers simply upload their code, and the platform executes it on their behalf as needed at any scale--is on the rise. This is because public cloud vendors have started offering new programming interfaces under the banner of serverless computing. The researchers support this with a Google search trend comparison where the term “serverless” recently matched the historic peak of popularity of the phrase “Map Reduce” or “MapReduce”. Source: arxiv.org They point out that the notion of serverless computing is vague enough to allow optimists to project any number of possible broad interpretations on what it might mean. Hence, in this paper, they have assessed the field based on the serverless computing services that vendors are actually offering today and also see why these services are a disappointment given that the cloud has a bigger potential. A Serverless architecture based on FaaS (Function-as-a-Service) Functions-as-a-Service (FaaS) is the commonly used and more descriptive name for the core of serverless offerings from the public cloud providers. Typical FaaS offerings today support a variety of languages (e.g., Python, Java, Javascript, Go), allow programmers to register functions with the cloud provider, and enable users to declare events that trigger each function. The FaaS infrastructure monitors the triggering events, allocates a runtime for the function, executes it, and persists the results. The user is billed only for the computing resources used during function invocation. Building applications on FaaS not only requires data management in both persistent and temporary storage but also mechanisms to trigger and scale function execution. According to the researchers, cloud providers are quick to emphasize that serverless is not only FaaS, but it is, FaaS supported by a “standard library”: the various multi-tenanted, autoscaling services provided by the vendor; for instance, S3 (large object storage), DynamoDB (key-value storage), SQS (queuing services), and more. However, current FaaS solutions are good for simple workloads of independent tasks such as parallel tasks embedded in Lambda functions, or jobs to be run by the proprietary cloud services. However, when it comes to use cases that involve stateful tasks, these FaaS have a surprisingly high latency. These realities limit the attractive use cases for FaaS today, discouraging new third-party programs that go beyond the proprietary service offerings from the vendors. Limitations of the current FaaS offering No recoverability Function invocations are shut down by the Lambda infrastructure automatically after 15 minutes. Lambda may keep the function’s state cached in the hosting VM in order to support a ‘warm start’ state. However, there is no way to ensure that subsequent invocations are run on the same VM. Hence functions must be written assuming that state will not be recoverable across invocations. I/O Bottlenecks Lambdas usually connect to cloud services or shared storage across a network interface. This means moving data across nodes or racks. With FaaS, things appear even worse than the network topology would suggest. Recent studies show that a single Lambda function can achieve on average 538 Mbps network bandwidth. This is an order of magnitude slower than a single modern SSD. Worse, AWS appears to attempt to pack Lambda functions from the same user together on a single VM, so the limited bandwidth is shared by multiple functions. The result is that as compute power scales up, per-function bandwidth shrinks proportionately. With 20 Lambda functions, average network bandwidth was 28.7Mbps—2.5 orders of magnitude slower than a single SSD. Communication Through Slow Storage Lambda functions can only communicate through an autoscaling intermediary service. As a corollary, a client of Lambda cannot address the particular function instance that handled the client’s previous request: there is no “stickiness” for client connections. Hence maintaining state across client calls require writing the state out to slow storage, and reading it back on every subsequent call. No Specialized Hardware FaaS offerings today only allow users to provision a time slice of a CPU hyperthread and some amount of RAM; in the case of AWS Lambda, one determines the other. There is no API or mechanism to access specialized hardware. These constraints, combined with some significant shortcomings in the standard library of FaaS offerings, substantially limit the scope of feasible serverless applications. The researchers conclude, “We see the future of cloud programming as far, far brighter than the promise of today’s serverless FaaS offerings. Getting to that future requires revisiting the designs and limitations of what is being called ‘serverless computing’ today.” They believe cloud programmers need to build a programmable framework that goes beyond FaaS, to dynamically manage the allocation of resources in order to meet user-specified performance goals for both compute and for data. The program analysis and scheduling issues are likely to open up significant opportunities for more formal research, especially for data-centric programs. To know more this research in detail, read the complete research paper. Introducing GitLab Serverless to deploy cloud-agnostic serverless functions and applications Introducing ‘Pivotal Function Service’ (alpha): an open, Kubernetes based, multi-cloud serverless framework for developer workloads Introducing numpywren, a system for linear algebra built on a serverless architecture
Read more
  • 0
  • 0
  • 5624

article-image-microsoft-becomes-the-worlds-most-valuable-public-company-moves-ahead-of-apple
Sugandha Lahoti
03 Dec 2018
3 min read
Save for later

Microsoft becomes the world's most valuable public company, moves ahead of Apple

Sugandha Lahoti
03 Dec 2018
3 min read
Last week, Microsoft moved ahead of Apple as the world’s most valuable publicly traded U.S. company. On Friday, the company closed on with a market value of $851 billion with Apple a few steps short at $847 billion. The move from Windows to Cloud Microsoft's success can be attributed to its able leadership under CEO Satya Nadella and his focus on moving away from the flagship Windows operating system and focusing on cloud-computing services with long-term business contracts. The organization's biggest growth has happened in its Azure Cloud platform. Cloud computing now accounts for more than a quarter of Microsoft’s revenue rivaling Amazon, which is also a leading provider. Microsoft is also building new products and features for Azure. Last month, it announced container support for Azure Cognitive Services to build intelligent applications. In October, it invested in Grab to together conquer the Southeast Asian on-demand services market with Azure’s Intelligent Cloud. In September, at the Ignite 2018, the company announced major changes and improvements to their cloud offering. It also came up with Azure Functions 2.0 with better workload support for serverless, general availability of Microsoft’s Immutable storage for Azure Storage Blobs, and Azure DevOps. In August, Microsoft made Azure supported for NVIDIA GPU Cloud (NGC), and a new governance DApp for Azure. Wedbush analyst Dan Ives commented that “Azure is still in its early days, meaning there’s plenty of room for growth, especially considering the company’s large customer base for Office and other products. While the tech carnage seen over the last month has been brutal, shares of (Microsoft) continue to hold up like the Rock of Gibraltar” he said. Focus on business and values Microsoft has also prioritized business-oriented services such as Office and other workplace software, as well as newer additions such as LinkedIn and Skype. In 2016, Microsoft bought LinkedIn, the social network for professionals, for $26.2 billion. This year, Microsoft paid $7.5 billion for GitHub, an open software platform used by 28 million programmers. Another reason Microsoft is flourishing is because of its focus on upholding its founding values without compromising on issues like internet censorship and surveillance. Daniel Morgan, senior portfolio manager for Synovus Trust, says “Microsoft is outperforming its tech rivals in part because it doesn’t face as much regulatory scrutiny as advertising-hungry Google and Facebook, which have attracted controversy over their data-harvesting practices. Unlike Netflix, it’s not on a hunt for a diminishing number of international subscribers. And while Amazon also has a strong cloud business, it’s still more dependent on online retail.” In a recent episode of Pivot with Kara Swisher and Scott Galloway, the two speakers also talked about why Microsoft is more valuable than Apple. Scott said that Microsoft’s success is because of Nadella’s decision of diversifying Microsoft’s business into enough verticals which is the reason why the company hasn’t been as impacted by tech stocks’ recent decline. He argues that Satya Nadella deserves the title of “tech CEO of the year”. Microsoft wins $480 million US Army contract for HoloLens. Microsoft amplifies focus on conversational AI: Acquires XOXCO; shares guide to developing responsible bots. Microsoft announces official support for Windows 10 to build 64-bit ARM apps
Read more
  • 0
  • 0
  • 3009

article-image-4-reasons-ibm-bought-red-hat-for-34-billion
Richard Gall
29 Oct 2018
8 min read
Save for later

4 reasons IBM bought Red Hat for $34 billion

Richard Gall
29 Oct 2018
8 min read
The news that IBM is to buy Red Hat - the enterprise Linux distribution - shocked the software world this weekend. It took many people by surprise because it signals a weird new world where the old guard of tech conglomerates - almost prehistoric in the history of the industry - are revitalizing themselves by diving deep into the open source world for pearls. So, why did IBM decide to buy Red Hat? And why has it spent so much to do it? Why did IBM decide to buy Red Hat? For IBM this was an expensive step into a new world. But they wouldn't have done it without good reason. And although it's hard to center on one single reason that forced IBM's decision makers to put money on the table, there are certainly a combination of factors that meant this move simply makes sense from IBM's perspective. Here are 4 reasons why IBM is buying Red Hat: Competing in the cloud market Disappointment around the success of IBM Watson Catching up with Microsoft To help provide support for an important but struggling Linux organization Let's take a look at each of these in more detail. IBM wants to get serious about cloud computing IBM has been struggling in a competitive cloud market. It's not exactly out of the running, with some reports placing them in third after AWS and Microsoft Azure, and others in fourth, with Google's cloud offering above them. But wherever the company stands, it's true that it is not growing at anywhere near the rate of its competitors. Put simply, if it didn't act, IBM would lose significant ground in the cloud computing race. It's no coincidence that cloud was right at the top of the IBM press release. Ginni Rometty, IBM Chairman, President and Chief Executive Officer, is quoted as saying "The acquisition of Red Hat is a game-changer. It changes everything about the cloud market... IBM will become the world's #1 hybrid cloud provider, offering companies the only open cloud solution that will unlock the full value of the cloud for their businesses." Clearly, IBM wants to bring itself up to date. As The Register wrote when they covered the story on Sunday IBM "really, really, really wants to transform itself into a cool and trendy hybrid cloud platform, rather than be seen eternally as a maintainer of legacy mainframes and databases." But why buy Red Hat? You might still be thinking well, why does IBM need Red Hat to do all this? Can't it just do it itself? It ultimately comes down to expanding what businesses can do with cloud - and bringing an open source company into the IBM family will allow IBM to deliver much more effectively on this than they have before. AWS appears to implicitly understand that features and capabilities are everything when it comes to cloud - to be truly successful, IBM needs to adopt both an open source mindset and toolset to innovate at a fast pace. This is what Rometty is referring to when she talks about "the next chapter of the cloud." This is where cloud becomes more about "extracting more data and optimizing every part of the business, from supply chains to sales" than storage space. IBM's artificial intelligence product, Watson, hasn't taken off IBM is a company with its proverbial finger in many pies. Its artificial intelligence product, Watson, hasn't had the success that the company expected. Instead, it has suffered a number of disappointing setbacks this year, resulting in Deborah DiSanzo, the head of Watson Health, stepping down just a week ago. One of the biggest stories was MD Anderson Cancer Center stepping away from a contract with IBM, after a report by analysts at investment bank Jeffries claimed that the software was "not ready for human investigational or clinical use." But there are other stories too - all of which come together to paint a picture of a project that doesn't live up to or deliver on its hype. By contrast, AI has been most impactful as a part of a cloud product. Just look at the furore around the AI tools within AWS - there's no way government agencies and the military would be quite so interested in the product if it wasn't packaged in a way that could be easily deployed. AWS, unlike IBM, understood that AI is only worth the hype if organizations can use it easily. In effect, we're past the period where AI deserves hype on its own - it needs to be part of a wider suite of capabilities that enable innovation and invention with minimal friction. If IBM is to offer out Watson's capabilities to a wide portion of users, all with varying use cases, IBM can begin to think much more about how the end product can deliver the biggest impact for these individual cases. IBM is playing catch up with Microsoft in terms of open source IBM's move might be surprising, but in the context of Microsoft's transformation over the last decade, it's part of a wider pattern. The only difference is that Microsoft's attitude to open source has slowly thawed, whereas IBM has gone all out, taking an unexpected leap into the unknown. It's a neat coincidence that this was the weekend that GitHub officially became part of Microsoft. It's as if IBM saw Microsoft basking in the glow of an open source embrace and thought we want that. Envy aside, there are serious implications. The future is now quite clearly open source - in fact, it has been for some time. You might even say that Microsoft hasn't been as quick as it could have been. But for IBM, open source has been seen simply as a tasty slice of the software pie - great, but not the whole thing. This was a misunderstanding - open source is everything. It almost doesn't even make sense to talk about open source as if it were distinctive from everything else - it is software today. It's defining the future. Joseph Jacks, the founder of Open Source Capital, said  that "IBM buying @RedHat is not about dominating the cloud. It is about becoming an OSS company. The largest proprietary software and tech companies in the world are now furiously rushing towards the future. An open future. An open source software driven future. OSS eats everything." https://twitter.com/asynchio/status/1056693588640194560   IBM is heavily invested in Linux - and RedHat isn't exactly thriving However, although open source might be the dominant mode of software in 2018, there are a few murmurs about it's sustainability and resilience. So, despite being central to just about everything we build and use when it comes to software, from a business perspective it isn't exactly thriving. Red Hat is a brilliant case in point. Despite being one of the first and most successful open source software businesses, providing free, open source software to customers in return for a support fee, revenues are down. Shares fell 14% in June following a disappointing financial forecast - and have fallen further since then. This piece in TechCrunch, almost 5 years old, does a good job of explaining the relative success of Red Hat, as well as its limitations: "When you compare the market cap and revenue of Red Hat to Microsoft or Amazon or Oracle, even Red Hat starts to look like a lukewarm success. The overwhelming success of Linux is disproportionate to the performance of Red Hat. Great for open source, a little disappointing for Red Hat." From this perspective, this sets the stage for an organisation like IBM to come in and start investing in Red Hat as a foundational component of its future product and software strategy. Given that both organizations are heavily invested in Linux, this could be a really important relationship in supporting the project in the future. And although a multi-billion acquisition might not look like open source in action, it might also be one of the only ways that it's going to survive and thrive in the future. Thanks to Amarabha Banerjee, Aarthi Kumaraswamy, and Amey Varangaonkar for their help with this post. Update on 9th July, 2019 As pert the reports from The Fortune, IBM on Tuesday morning closed its $34 billion acquisition of Red Hat, which was announced last October. The pricey deal, which paid Red Hat owners a hefty premium of more than 60%, marks IBM CEO Ginni Rometty’s biggest bet yet in transforming her 108-year-old technology company. In an interview Tuesday morning, she said some tech analysts have assumed the move to the cloud would lead to a “winner take all” scenario, where one giant platform—Amazon Web Services?—ends up with all the business. Read the full story here.
Read more
  • 0
  • 0
  • 12901
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at $19.99/month. Cancel anytime
article-image-ibm-acquired-red-hat-for-34-billion-making-it-the-biggest-open-source-acquisition-ever
Sugandha Lahoti
29 Oct 2018
4 min read
Save for later

IBM acquired Red Hat for $34 billion making it the biggest open-source acquisition ever

Sugandha Lahoti
29 Oct 2018
4 min read
In probably the biggest open source acquisition ever, IBM has acquired all of the issued and outstanding common shares of Red Hat for $190.00 per share in cash, representing a total enterprise value of approximately $34 billion. However, if this deal is more of a business proposition than a community contributor is a question. Red Hat has been struggling on the market recently. Red Hat missed its most recent revenue estimates and its guidance fell below Wall Street targets. Prior to this deal, it had a market capitalization of about $20.5 billion. With this deal, Red Hat may soon take control of it’s sinking ship. It will also remain a distinct unit within IBM. The company will continue to be led by Jim Whitehurst, Red Hat’s CEO and Red Hat's current management team. Jim Whitehurst also will join IBM's senior management team and report to Ginni Rometty, IBM Chairman, President, and Chief Executive Officer. Why is Red Hat joining forces with IBM? In the announcement, Jim assured that IBM’s acquisition of Red Hat will help them accelerate without compromising their culture and policies. He said, "Open source is the default choice for modern IT solutions, and I'm incredibly proud of the role Red Hat has played in making that a reality in the enterprise.” He also added that, “Joining forces with IBM will provide us with a greater level of scale, resources, and capabilities to accelerate the impact of open source as the basis for digital transformation and bring Red Hat to an even wider audience--all while preserving our unique culture and unwavering commitment to open source innovation." What is IBM gaining from this acquisition? IBM believes this acquisition to be a game changer. "It changes everything about the cloud market," said Ginni, "IBM will become the world's #1 hybrid cloud provider, offering companies the only open cloud solution that will unlock the full value of the cloud for their businesses. IBM and Red Hat will accelerate hybrid multi-cloud adoption across all companies. They plan to together, “help clients create cloud-native business applications faster, drive greater portability and security of data and applications across multiple public and private clouds, all with consistent cloud management.” "IBM is committed to being an authentic multi-cloud provider, and we will prioritize the use of Red Hat technology across multiple clouds," said Arvind Krishna, Senior Vice President, IBM Hybrid Cloud. "In doing so, IBM will support open source technology wherever it runs, allowing it to scale significantly within commercial settings around the world." IBM assures that it will continue to build and enhance Red Hat partnerships with major cloud providers. It will also remain committed to Red Hat's open governance, open source contributions, participation in the open source community and development model. The company is keen on preserving the independence and neutrality of Red Hat's open source development culture and go-to-market strategy. The news was well received by the top Red Hat decision makers who embraced this with open arms. However, ZDNet reported that many RedHat employees were skeptical: "I can't imagine a bigger culture clash." "I'll be looking for a job with an open-source company." "As a Red Hat employee, almost everyone here would prefer it if we were bought out by Microsoft." People’s reactions on twitter on this acquisition are also varied: https://twitter.com/samerkamal/status/1056611186584604672 https://twitter.com/pnuojua/status/1056787520845955074 https://twitter.com/CloudStrategies/status/1056666824434020352 https://twitter.com/svenpet/status/1056646295002247169 Read more about the news on IBM’s newsroom. Red Hat infrastructure migration solution for proprietary and siloed infrastructure. IBM launches Industry’s first ‘Cybersecurity Operations Center on Wheels’ for on-demand cybersecurity support IBM Watson announces pre-trained AI tools to accelerate IoT operations
Read more
  • 0
  • 0
  • 2965

article-image-chaos-conf-2018-recap-chaos-engineering-hits-maturity-as-community-moves-towards-controlled-experimentation
Richard Gall
12 Oct 2018
11 min read
Save for later

Chaos Conf 2018 Recap: Chaos engineering hits maturity as community moves towards controlled experimentation

Richard Gall
12 Oct 2018
11 min read
Conferences can sometimes be confusing. Even at the most professional and well-planned conferences, you sometimes just take a minute and think what's actually the point of this? Am I learning anything? Am I meant to be networking? Will anyone notice if I steal extra food for the journey home? Chaos Conf 2018 was different, however. It had a clear purpose: to take the first step in properly forging a chaos engineering community. After almost a decade somewhat hidden in the corners of particularly innovative teams at Netflix and Amazon, chaos engineering might feel that its time has come. As software infrastructure becomes more complex, less monolithic, and as business and consumer demands expect more of the software systems that have become integral to the very functioning of life, resiliency has never been more important but more challenging to achieve. But while it feels like the right time for chaos engineering, it hasn't quite established itself in the mainstream. This is something the conference host, Gremlin, a platform that offers chaos engineering as a service, is acutely aware of. On the hand it's actively helping push chaos engineering into the hands of businesses, but on the other its growth and success, backed by millions of VC cash (and faith), depends upon chaos engineering becoming a mainstream discipline in the DevOps and SRE worlds. It's perhaps this reason that the conference felt so important. It was, according to Gremlin, the first ever public chaos engineering conference. And while it was relatively small in the grand scheme of many of today's festival-esque conferences attended by thousands of delegates (Dreamforce, the Salesforce conference, was also running in San Francisco in the same week), the fact that the conference had quickly sold out all 350 of its tickets - with more hoping on waiting lists - indicates that this was an event that had been eagerly awaited. And with some big names from the industry - notably Adrian Cockcroft from AWS and Jessie Frazelle from Microsoft - Chaos Conf had the air of an event that had outgrown its insider status before it had even began. The renovated cinema and bar in San Francisco's Mission District, complete with pinball machines upstairs, was the perfect container for a passionate community that had grown out of the clean corporate environs of Silicon Valley to embrace the chaotic mess that resembles modern software engineering. Kolton Andrus sets out a vision for the future of Gremlin and chaos engineering Chaos Conf was quick to deliver big news. They keynote speech, by Gremlin co-founder Kolton Andrus launched Gremlin's brand new Application Level Fault Injection (ALFI) feature, which makes it possible to run chaos experiments at an application level. Andrus broke the news by building towards it with a story of the progression of chaos engineering. Starting with Chaos Monkey, the tool first developed by Netflix, and moving from infrastructure to network, he showed how, as chaos engineering has evolved, it requires and faciliates different levels of control and insight on how your software works. "As we're maturing, the host level failures and the network level failures are necessary to building a robust and resilient system, but not sufficient. We need more - we need a finer granularity," Andrus explains. This is where ALFI comes in. By allowing Gremlin users to inject failure at an application level, it allows them to have more control over the 'blast radius' of their chaos experiments. The narrative Andrus was setting was clear, and would ultimately inform the ethos of the day - chaos engineering isn't just about chaos, it's about controlled experimentation to ensure resiliency. To do that requires a level of intelligence - technical and organizational - about how the various components of your software work, and how humans interact with them. Adrian Cockcroft on the importance of historical context and domain knowledge Adrian Cockcroft (@adrianco) VP at AWS followed Andrus' talk. In it he took the opportunity to set the broader context of chaos engineering, highlighting how tackling system failures is often a question of culture - how we approach system failure and think about our software. Developers love to learn things from first principles" he said. "But some historical context and domain knowledge can help illuminate the path and obstacles." If this sounds like Cockcroft was about to stray into theoretical territory, he certainly didn't. He offered plenty of practical frameworks for thinking through potential failure. But the talk wasn't theoretical - Cockcroft offered a taxonomy of failure that provides a useful framework for thinking through potential failure at every level. He also touched on how he sees the future of resiliency evolving, focusing on: Observability of systems Epidemic failure modes Automation and continuous chaos The crucial point Cockcroft makes is that cloud is the big driver for chaos engineering. "As datacenters migrate to the cloud, fragile and manual disaster recovery will be replaced by chaos engineering" read one of his slides. But more than that, the cloud also paves the way for the future of the discipline, one where 'chaos' is simply an automated part of the test and deployment pipeline. Selling chaos engineering to your boss Kriss Rochefolle, DevOps engineer and author of one of the best selling DevOps books in French, delivered a short talk on how engineers can sell chaos to their boss. He takes on the assumption that a rational proposal, informed by ROI is the best way to sell chaos engineering. He suggests instead that engineers need to play into emotions, and presenting chaos engineer as a method for tackling and minimizing the fear of (inevitable failure. Follow Kriss on Twitter: @crochefolle Walmart and chaos engineering Vilas Veraraghavan, the Director of Engineering was keen to clarify that Walmart doesn't practice chaos. Rather it practices resiliency - chaos engineering is simply a method the organization uses to achieve that. It was particularly useful to note the entire process that Vilas' team adopts when it comes to resiliency, which has largely developed out of Vilas' own work building his team from scratch. You can learn more about how Walmart is using chaos engineering for software resiliency in this post. Twitter's Ronnie Chen on diving and planning for failure Ronnie Chen (@rondoftw) is an engineering manager at Twitter. But she didn't talk about Twitter. In fact, she didn't even talk about engineering. Instead she spoke about her experience as a technical diver. By talking about her experiences, Ronnie was able to make a number of vital points about how to manage and tackle failure as a team. With mortality rates so high in diving, it's a good example of the relationship between complexity and risk. Chen made the point that things don't fail because of a single catalyst. Instead, failures - particularly fatal ones - happen because of a 'failure cascade'. Chen never makes the link explicit, but the comparison is clear - the ultimate outcome (ie. success or failure) is impacted by a whole range of situational and behavioral factors that we can't afford to ignore. Chen also made the point that, in diving, inexperienced people should be at the front of an expedition. "If you're inexperienced people are leading, they're learning and growing, and being able to operate with a safety net... when you do this, all kinds of hidden dependencies reveal themselves... every undocumented assumption, every piece of ancient team lore that you didn't even know you were relying on, comes to light." Charity Majors on the importance of observability Charity Majors (@mipsytipsy), CEO of Honeycomb, talked in detail about the key differences between monitoring and observability. As with other talks, context was important: a world where architectural complexity has grown rapidly in the space of a decade. Majors made the point that this increase in complexity has taken us from having known unknowns in our architectures, to many more unknown unknowns in a distributed system. This means that monitoring is dead - it simply isn't sophisticated enough to deal with the complexities and dependencies within a distributed system. Observability, meanwhile, allows you to to understand "what's happening in your systems just by observing it from the outside." Put simply, it lets you understand how your software is functioning from your perspective - almost turning it inside out. Majors then linked the concept to observability to the broader philosophy of chaos engineering - echoing some of the points raised by Adrian Cockcroft in his keynote. But this was her key takeaway: "Software engineers spend too much time looking at code in elaborately falsified environments, and not enough time observing it in the real world." This leads to one conclusion - the importance of testing in production. "Accept no substitute." Tammy Butow and Ana Medina on making an impact Tammy Butow (@tammybutow) and Ana Medina  (@Ana_M_Medina) from Gremlin took us through how to put chaos engineering into practice - from integrating it into your organizational culture to some practical tests you can run. One of the best examples of putting chaos into practice is Gremlin's concept of 'Failure Fridays', in which chaos testing becomes a valuable step in the product development process, to dogfood it and test out how a customer experiences it. Another way which Tammy and Ana suggested chaos engineering can be used was as a way of testing out new versions of technologies before you properly upgrade in production. To end, their talk, they demo'd a chaos battle between EKS (Kubernetes on AWS) and AKS (Kubernetes on Azure), doing an app container attack, a packet loss attack and a region failover attack. Jessie Frazelle on how containers can empower experimentation Jessie Frazelle (@jessfraz) didn't actually talk that much about chaos engineering. However, like Ronnie Chen's talk, chaos engineering seeped through what she said about bugs and containers. Bugs, for Frazelle, are a way of exploring how things work, and how different parts of a software infrastructure interact with each other: "Bugs are like my favorite thing... some people really hate when they get one of those bugs that turns out to be a rabbit hole and your kind of debugging it until the end of time... while debugging those bugs I hate them but afterwards, I'm like, that was crazy!" This was essentially an endorsement of the core concept of chaos engineering - injecting bugs into your software to understand how it reacts. Jessie then went on to talk about containers, joking that they're NOT REAL. This is because they're made up of  numerous different component parts, like Cgroups, namespaces, and LSMs. She contrasted containers with Virtual machines, zones and jails, which are 'first class concepts' - in other worlds, real things (Jessie wrote about this in detail last year in this blog post). In practice what this means is that whereas containers are like Lego pieces, VMs, zones, and jails are like a pre-assembled lego set that you don't need to play around with in the same way. From this perspective, it's easy to see how containers are relevant to chaos engineering - they empower a level of experimentation that you simply don't have with other virtualization technologies. "The box says to build the death star. But you can build whatever you want." The chaos ends... Chaos Conf was undoubtedly a huge success, and a lot of credit has to go to Gremlin for organizing the conference. It's clear that the team care a lot about the chaos engineering community and want it to expand in a way that transcends the success of the Gremlin platform. While chaos engineering might not feel relevant to a lot of people at the moment, it's only a matter of time that it's impact will be felt. That doesn't mean that everyone will suddenly become a chaos engineer by July 2019, but the cultural ripples will likely be felt across the software engineering landscape. But without Chaos Conf, it would be difficult to see chaos engineering growing as a discipline or set of practices. By sharing ideas and learning how other people work, a more coherent picture of chaos engineering started to emerge, one that can quickly make an impact in ways people wouldn't have expected six months ago. You can watch videos of all the talks from Chaos Conf 2018 on YouTube.
Read more
  • 0
  • 0
  • 5525

article-image-cloud-native-architectures-microservices-containers-serverless-part-2
Guest Contributor
14 Aug 2018
8 min read
Save for later

Modern Cloud Native architectures: Microservices, Containers, and Serverless - Part 2

Guest Contributor
14 Aug 2018
8 min read
This whitepaper is written by Mina Andrawos, an experienced engineer who has developed deep experience in the Go language, and modern software architectures. He regularly writes articles and tutorials about the Go language, and also shares open source projects. Mina Andrawos has authored the book Cloud Native programming with Golang, which provides practical techniques, code examples, and architectural patterns required to build cloud native microservices in the Go language.He is also the author of the Mastering Go Programming, and the Modern Golang Programming video courses. We published Part 1 of this paper yesterday and here we come up with Part 2 which involves Containers and Serverless applications. Let us get started: Containers The technology of software containers is the next key technology that needs to be discussed to practically explain cloud native applications. A container is simply the idea of encapsulating some software inside an isolated user space or “container.” For example, a MySQL database can be isolated inside a container where the environmental variables, and the configurations that it needs will live. Software outside the container will not see the environmental variables or configuration contained inside the container by default. Multiple containers can exist on the same local virtual machine, cloud virtual machine, or hardware server. Containers provide the ability to run numerous isolated software services, with all their configurations, software dependencies, runtimes, tools, and accompanying files, on the same machine. In a cloud environment, this ability translates into saved costs and efforts, as the need for provisioning and buying server nodes for each microservices will diminish, since different microservices can be deployed on the same host without disrupting each other. Containers  combined with microservices architectures are powerful tools to build modern, portable, scalable, and cost efficient software. In a production environment, more than a single server node combined with numerous containers would be needed to achieve scalability and redundancy. Containers also add more benefits to cloud native applications beyond microservices isolation. With a container, you can move your microservices, with all the configuration, dependencies, and environmental variables that it needs, to fresh server nodes without the need to reconfigure the environment, achieving powerful portability. Due to the power and popularity of the software containers technology, some new operating systems like CoreOS, or Photon OS, are built from the ground up to function as hosts for containers. One of the most popular software container projects in the software industry is Docker. Major organizations such as Cisco, Google, and IBM utilize Docker containers in their infrastructure as well as in their products. Another notable project in the software containers world is Kubernetes. Kubernetes is a tool that allows the automation of deployment, management, and scaling of containers. It was built by Google to facilitate the management of their containers, which are counted by billions per week. Kubernetes provides some powerful features such as load balancing between containers, restart for failed containers, and orchestration of storage utilized by the containers. The project is part of the cloud native foundation along with Prometheus. Container complexities In case of containers, sometimes the task of managing them can get rather complex for the same reasons as managing expanding numbers of microservices. As containers or microservices grow in size, there needs to be a mechanism to identify where each container or microservices is deployed, what their purpose is, and what they need in resources to keep running. Serverless applications Serverless architecture is a new software architectural paradigm that was popularized with the AWS Lambda service. In order to fully understand serverless applications, we must first cover an important concept known as ‘Function As A service’, or FaaS for short. Function as a service or FaaS is the idea that a cloud provider such as Amazon or even a local piece of software such as Fission.io or funktion would provide a service, where a user can request a function to run remotely in order to perform a very specific task, and then after the function concludes, the function results return back to the user. No services or stateful data are maintained and the function code is provided by the user to the service that runs the function. The idea behind properly designed cloud native production applications that utilize the serverless architecture is that instead of building multiple microservices expected to run continuously in order to carry out individual tasks, build an application that has fewer microservices combined with FaaS, where FaaS covers tasks that don’t need services to run continuously. FaaS is a smaller construct than a microservice. For example, in case of the event booking application we covered earlier, there were multiple microservices covering different tasks. If we use a serverless applications model, some of those microservices would be replaced with a number of functions that serve their purpose. Here is a diagram that showcases the application utilizing a serverless architecture: In this diagram, the event handler microservices as well as the booking handler microservices were replaced with a number of functions that produce the same functionality. This eliminates the need to run and maintain the two existing microservices. Serverless architectures have the advantage that no virtual machines and/or containers need to be provisioned to build the part of the application that utilizes FaaS. The computing instances that run the functions cease to exist from the user point of view once their functions conclude. Furthermore, the number of microservices and/or containers that need to be monitored and maintained by the user decreases, saving cost, time, and effort. Serverless architectures provide yet another powerful software building tool in the hands of software engineers and architects to design flexible and scalable software. Known FaaS are AWS Lambda by Amazon, Azure Functions by Microsoft, Cloud Functions by Google, and many more. Another definition for serverless applications is the applications that utilize the BaaS or backend as a service paradigm. BaaS is the idea that developers only write the client code of their application, which then relies on several software pre-built services hosted in the cloud, accessible via APIs. BaaS is popular in mobile app programming, where developers would rely on a number of backend services to drive the majority of the functionality of the application. Examples of BaaS services are: Firebase, and Parse. Disadvantages of serverless applications Similarly to microservices and cloud native applications, the serverless architecture is not suitable for all scenarios. The functions provided by FaaS don’t keep state by themselves which means special considerations need to be observed when writing the function code. This is unlike a full microservice, where the developer has full control over the state. One approach to keep state in case of FaaS, in spite of this limitation, is to propagate the state to a database or a memory cache like Redis. The startup times for the functions are not always fast since there is time allocated to sending the request to the FaaS service provider then the time needed to start a computing instance that runs the function in some cases. These delays have to be accounted for when designing serverless applications. FaaS do not run continuously like microservices, which makes them unsuitable for any task that requires continuous running of the software. Serverless applications have the same limitation as other cloud native applications where portability of the application from one cloud provider to another or from the cloud to a local environment becomes challenging because of vendor lock-in Conclusion Cloud computing architectures have opened avenues for developing efficient, scalable, and reliable software. This paper covered some significant concepts in the world of cloud computing such as microservices, cloud native applications, containers, and serverless applications. Microservices are the building blocks for most scalable cloud native applications; they decouple the application tasks into various efficient services. Containers are how microservices could be isolated and deployed safely to production environments without polluting them.  Serverless applications decouple application tasks into smaller constructs mostly called functions that can be consumed via APIs. Cloud native applications make use of all those architectural patterns to build scalable, reliable, and always available software. You read Part 2 of of Modern cloud native architectures, a white paper by Mina Andrawos. Also read Part 1 which includes Microservices and Cloud native applications with their advantages and disadvantages. If you are interested to learn more, check out Mina’s Cloud Native programming with Golang to explore practical techniques for building cloud-native apps that are scalable, reliable, and always available. About Author: Mina Andrawos Mina Andrawos is an experienced engineer who has developed deep experience in Go from using it personally and professionally. He regularly authors articles and tutorials about the language, and also shares Go's open source projects. He has written numerous Go applications with varying degrees of complexity. Other than Go, he has skills in Java, C#, Python, and C++. He has worked with various databases and software architectures. He is also skilled with the agile methodology for software development. Besides software development, he has working experience of scrum mastering, sales engineering, and software product management. Build Java EE containers using Docker [Tutorial] Are containers the end of virtual machines? Why containers are driving DevOps
Read more
  • 0
  • 0
  • 3950

article-image-cloud-native-architectures-microservices-containers-serverless-part-1
Guest Contributor
13 Aug 2018
9 min read
Save for later

Modern Cloud Native architectures: Microservices, Containers, and Serverless - Part 1

Guest Contributor
13 Aug 2018
9 min read
This whitepaper is written by Mina Andrawos, an experienced engineer who has developed deep experience in the Go language, and modern software architectures. He regularly writes articles and tutorials about the Go language, and also shares open source projects. Mina Andrawos has authored the book Cloud Native programming with Golang, which provides practical techniques, code examples, and architectural patterns required to build cloud native microservices in the Go language.He is also the author of the Mastering Go Programming, and the Modern Golang Programming video courses. This paper sheds some light and provides practical exposure on some key topics in the modern software industry, namely cloud native applications.This includes microservices, containers , and serverless applications. The paper will cover the practical advantages, and disadvantages of the technologies covered. Microservices The microservices architecture has gained reputation as a powerful approach to architect modern software applications. So what are microservices? Microservices can be described as simply the idea of separating the functionality required from a software application into multiple independent small software services or “microservices.” Each microservice is responsible for an individual focused task. In order for microservices to collaborate together to form a large scalable application, they communicate and exchange data. Microservices were born out of the need to tame the complexity, and inflexibility of “monolithic” applications. A monolithic application is a type of application, where all required functionality is coded together into the same service. For example, here is a diagram representing a monolithic events (like concerts, shows..etc) booking application that takes care of the booking payment processing and event reservation: The application can be used by a customer to book a concert or a show. A user interface will be needed. Furthermore, we will also need a search functionality to look for events, a bookings handler to process the user booking then save it, and an events handler to help find the event, ensure it has seats available, then link it to the booking. In a production level application, more tasks will be needed like payment processing for example, but for now let’s focus on the four tasks outlined in the above figure. This monolithic application will work well with small to medium load. It will run on a single server, connect to a single database and will be written probably in the same programming language. Now, what will happen if the business grows exponentially and hundreds of thousands or millions of users need to be handled and processed? Initially, the short term solution would be to ensure that the server where the application runs, has powerful hardware specifications to withstand higher loads, and if not then add more memory, storage, and processing power to the server. This is called vertical scaling, which is the act of increasing the power of the hardware  like RAM and hard drive capacity to run heavy applications.However, this is typically not  sustainable in the long run as the load on the application continues to grow. Another challenge with monolithic applications is the inflexibility caused by being limited to only one or two programming languages. This inflexibility can affect the overall quality, and efficiency of the application. For example, node.js is a popular JavaScript framework for building web applications, whereas R is popular for data science applications. A monolithic application will make it difficult to utilize both technologies, whereas in a microservices application, we can simply build a data science service written in R and a web service written in Node.js. The microservices version of the events application will take the below form: This application will be capable of scaling among multiple servers, a practice known as horizontal scaling. Each service can be deployed on a different server with dedicated resources or in separate containers (more on that later). The different services can be written in different programming languages enabling greater flexibility, and different dedicated teams can focus on different services achieving more overall quality for the application. Another notable advantage of using microservices is the ease of continuous delivery, which is the ability to deploy software often, and at any time. The reason why microservices make continuous delivery easier is because a new feature deployed to one microservices is less likely to affect other microservices compared to monolithic applications. Issues with Microservices One notable drawback of relying heavily on microservices is the fact that they can become too complicated to manage in the long run as they grow in numbers and scope. There are approaches to mitigate this by utilizing monitoring tools such as Prometheus to detect problems, container technologies such as Docker to avoid pollutions of the host environments and avoiding over designing the services. However, these approaches take effort and time. Cloud native applications Microservices architectures are a natural fit for cloud native applications. A cloud native application is simply defined as an application built from the ground up for cloud computing architectures. This simply means that our application is cloud native, if we design it as if it is expected to be deployed on a distributed, and scalable infrastructure. For example, building an application with a redundant microservices architecture -we’ll see an example shortly- makes the application cloud native, since this architecture allows our application to be deployed in a distributed manner that allows it to be scalable and almost always available. A cloud native application does not need to always be deployed to a public cloud like AWS, we can deploy it to our own distributed cloud-like infrastructure instead if we have one. In fact, what makes an application fully cloud native is beyond just using microservices. Your application  should employ continuous delivery, which is your ability to continuously deliver updates to your production applications without  disruptions. Your application should also make use of services like message queues and technologies like containers, and serverless (containers and serverless are important topics for modern software architectures, so we’ll be discussing them in the next few sections). Cloud native applications assume access to numerous server nodes, having access to pre-deployed software services like message queues or load balancers, ease of integration with continuous delivery services, among other things. If you deploy your cloud native application to a commercial cloud like AWS or Azure, your application gets the option to utilize cloud only software services. For example, DynamoDB is a powerful database engine that can only be used on Amazon Web Services for production applications. Another example is the DocumentDB database in Azure. There are also cloud only message queues such as Amazon Simple Queue Service (SQS), which can be used to allow communication between microservices in the Amazon Web Services cloud. As mentioned earlier, cloud native microservices should be designed to allow redundancy between services. If we take the events booking application as an example, the application will look like this: Multiple server nodes would be allocated per microservice, allowing a redundant microservices architecture to be deployed. If the primary node or service fails for any reason, the secondary can take over ensuring lasting reliability and availability for cloud native applications. This availability is vital for fault intolerant applications such as e-commerce platforms, where downtime translates into large amounts of lost revenue. Cloud native applications provide great value for developers, enterprises, and startups. A notable tool worth mentioning in the world of microservices and cloud computing is Prometheus. Prometheus is an open source system monitoring and alerting tool that can be used to monitor complex microservices architectures and alert when an action needs to be taken. Prometheus was originally created by SoundCloud to monitor their systems, but then grew to become an independent project. The project is now a part of the cloud native computing foundation, which is a foundation tasked with building a sustainable ecosystem for cloud native applications. Cloud native limitations For cloud native applications, you will face some challenges if the need arises to migrate some or all of the applications. That is due to multiple reasons, depending on where your application is deployed. For example,if your cloud native application is deployed on a public cloud like AWS, cloud native APIs are not cross cloud platform. So, a DynamoDB database API utilized in an application will only work on AWS but not on Azure, since DynamoDB belongs exclusively to AWS. The API will also never work in a local environment because DynamoDB can only be utilized in AWS in production. Another reason is because there are some assumptions made when some cloud native applications are built, like the fact that there will be virtually unlimited number of server nodes to utilize when needed and that a new server node can be made available very quickly. These assumptions are sometimes hard to guarantee in a local data center environment, where real servers, networking hardware, and wiring need to be purchased. This brings us to the end of Part 1 of this whitepaper. Check out Part 2 tomorrow to learn about Containers and Serverless applications along with their practical advantages and limitations. About Author: Mina Andrawos Mina Andrawos is an experienced engineer who has developed deep experience in Go from using it personally and professionally. He regularly authors articles and tutorials about the language, and also shares Go's open source projects. He has written numerous Go applications with varying degrees of complexity. Other than Go, he has skills in Java, C#, Python, and C++. He has worked with various databases and software architectures. He is also skilled with the agile methodology for software development. Besides software development, he has working experience of scrum mastering, sales engineering, and software product management. Building microservices from a monolith Java EE app [Tutorial] 6 Ways to blow up your Microservices! Have Microservices killed the monolithic architecture? Maybe not!
Read more
  • 0
  • 0
  • 3906
article-image-ansible-2-automate-networking-tasks-on-google-cloud
Vijin Boricha
31 Jul 2018
8 min read
Save for later

Ansible 2 for automating networking tasks on Google Cloud Platform [Tutorial]

Vijin Boricha
31 Jul 2018
8 min read
Google Cloud Platform is one of the largest and most innovative cloud providers out there. It is used by various industry leaders such as Coca-Cola, Spotify, and Philips. Amazon Web Services and Google Cloud are always involved in a price war, which benefits consumers greatly. Google Cloud Platform covers 12 geographical regions across four continents with new regions coming up every year. In this tutorial, we will learn about Google compute engine and network services and how Ansible 2 can be leveraged to automate common networking tasks. This is an excerpt from Ansible 2 Cloud Automation Cookbook written by Aditya Patawari, Vikas Aggarwal.  Managing network and firewall rules By default, inbound connections are not allowed to any of the instances. One way to allow the traffic is by allowing incoming connections to a certain port of instances carrying a particular tag. For example, we can tag all the webservers as http and allow incoming connections to port 80 and 8080 for all the instances carrying the http tag. How to do it… We will create a firewall rule with source tag using the gce_net module: - name: Create Firewall Rule with Source Tags gce_net: name: my-network fwname: "allow-http" allowed: tcp:80,8080 state: "present" target_tags: "http" subnet_region: us-west1 service_account_email: "{{ service_account_email }}" project_id: "{{ project_id }}" credentials_file: "{{ credentials_file }}" tags: - recipe6 Using tags for firewalls is not possible all the time. A lot of organizations whitelist internal IP ranges or allow office IPs to reach the instances over the network. A simple way to allow a range of IP addresses is to use a source range: - name: Create Firewall Rule with Source Range gce_net: name: my-network fwname: "allow-internal" state: "present" src_range: ['10.0.0.0/16'] subnet_name: public-subnet allowed: 'tcp' service_account_email: "{{ service_account_email }}" project_id: "{{ project_id }}" credentials_file: "{{ credentials_file }}" tags: - recipe6 How it works... In step 1, we have created a firewall rule called allow-http to allow incoming requests to TCP port 80 and 8080. Since our instance app is tagged with http, it can accept incoming traffic to port 80 and 8080. In step 2, we have allowed all the instances with IP 10.0.0.0/16, which is a private IP address range. Along with connection parameters and the source IP address CIDR, we have defined the network name and subnet name. We have allowed all TCP connections. If we want to restrict it to a port or a range of ports, then we can use tcp:80 or tcp:4000-5000 respectively. Managing load balancer An important reason to use a cloud is to achieve scalability at a relatively low cost. Load balancers play a key role in scalability. We can attach multiple instances behind a load balancer to distribute the traffic between the instances. Google Cloud load balancer also supports health checks which helps to ensure that traffic is sent to healthy instances only. How to do it… Let us create a load balancer and attach an instance to it: - name: create load balancer and attach to instance gce_lb: name: loadbalancer1 region: us-west1 members: ["{{ zone }}/app"] httphealthcheck_name: hc httphealthcheck_port: 80 httphealthcheck_path: "/" service_account_email: "{{ service_account_email }}" project_id: "{{ project_id }}" credentials_file: "{{ credentials_file }}" tags: - recipe7 For creating a load balancer, we need to supply a comma separated list of instances. We also need to provide health check parameters including a name, a port and the path on which a GET request can be sent. Managing GCE images in Ansible 2 Images are a collection of a boot loader, operating system, and a root filesystem. There are public images provided by Google and various open source communities. We can use these images to create an instance. GCE also provides us capability to create our own image which we can use to boot instances. It is important to understand the difference between an image and a snapshot. A snapshot is incremental but it is just a disk snapshot. Due to its incremental nature, it is better for creating backups. Images consist of more information such as a boot loader. Images are non-incremental in nature. However, it is possible to import images from a different cloud provider or datacenter to GCE. Another reason we recommend snapshots for backup is that taking a snapshot does not require us to shut down the instance, whereas building an image would require us to shut down the instance. Why build images at all? We will discover that in subsequent sections. How to do it… Let us create an image for now: - name: stop the instance gce: instance_names: app zone: "{{ zone }}" machine_type: f1-micro image: centos-7 state: stopped service_account_email: "{{ service_account_email }}" credentials_file: "{{ credentials_file }}" project_id: "{{ project_id }}" disk_size: 15 metadata: "{{ instance_metadata }}" tags: - recipe8 - name: create image gce_img: name: app-image source: app zone: "{{ zone }}" state: present service_account_email: "{{ service_account_email }}" pem_file: "{{ credentials_file }}" project_id: "{{ project_id }}" tags: - recipe8 - name: start the instance gce: instance_names: app zone: "{{ zone }}" machine_type: f1-micro image: centos-7 state: started service_account_email: "{{ service_account_email }}" credentials_file: "{{ credentials_file }}" project_id: "{{ project_id }}" disk_size: 15 metadata: "{{ instance_metadata }}" tags: - recipe8 How it works... In these tasks, we are stopping the instance first and then creating the image. We just need to supply the instance name while creating the image, along with the standard connection parameters. Finally, we start the instance back. The parameters of these tasks are self-explanatory. Creating instance templates Instance templates define various characteristics of an instance and related attributes. Some of these attributes are: Machine type (f1-micro, n1-standard-1, custom) Image (we created one in the previous tip, app-image) Zone (us-west1-a) Tags (we have a firewall rule for tag http) How to do it… Once a template is created, we can use it to create a managed instance group which can be auto-scale based on various parameters. Instance templates are typically available globally as long as we do not specify a restrictive parameter like a specific subnet or disk: - name: create instance template named app-template gce_instance_template: name: app-template size: f1-micro tags: http,http-server image: app-image state: present subnetwork: public-subnet subnetwork_region: us-west1 service_account_email: "{{ service_account_email }}" credentials_file: "{{ credentials_file }}" project_id: "{{ project_id }}" tags: - recipe9 We have specified the machine type, image, subnets, and tags. This template can be used to create instance groups. Creating managed instance groups Traditionally, we have managed virtual machines individually. Instance groups let us manage a group of identical virtual machines as a single entity. These virtual machines are created from an instance template, like the one which we created in the previous tip. Now, if we have to make a change in instance configuration, that change would be applied to all the instances in the group. How to do it… Perhaps, the most important feature of an instance group is auto-scaling. In event of high resource requirements, the instance group can scale up to a predefined number automatically: - name: create an instance group with autoscaling gce_mig: name: app-mig zone: "{{ zone }}" service_account_email: "{{ service_account_email }}" credentials_file: "{{ credentials_file }}" project_id: "{{ project_id }}" state: present size: 2 named_ports: - name: http port: 80 template: app-template autoscaling: enabled: yes name: app-autoscaler policy: min_instances: 2 max_instances: 5 cool_down_period: 90 cpu_utilization: target: 0.6 load_balancing_utilization: target: 0.8 tags: - recipe10 How it works... The preceding task creates an instance group with an initial size of two instances, defined by size. We have named port 80 as HTTP. This can be used by other GCE components to route traffic. We have used the template that we created in the previous recipe. We also enable autoscaling with a policy to allow scaling up to five instances. At any given point, at least two instances would be running. We are scaling on two parameters, cpu_utilization, where 0.6 would trigger scaling after the utilization exceeds 60% and load_balancing_utilization where the scaling will trigger after 80% of the requests per minutes capacity is reached. Typically, when an instance is booted, it might take some time for initialization and startup. Data collected during that period might not make much sense. The parameter, cool_down_period, indicates that we should start collecting data from the instance after 90 seconds and should not trigger scaling based on data before. We learnt a few networking tricks to manage public cloud infrastructure effectively. You can know more about building the public cloud infrastructure by referring to this book Ansible 2 Cloud Automation Cookbook. Why choose Ansible for your automation and configuration management needs? Getting Started with Ansible 2 Top 7 DevOps tools in 2018
Read more
  • 0
  • 0
  • 4259

article-image-googles-event-driven-serverless-platform-cloud-function-is-now-generally-available
Vijin Boricha
26 Jul 2018
2 min read
Save for later

Google's event-driven serverless platform, Cloud Function, is now generally available

Vijin Boricha
26 Jul 2018
2 min read
Early this week, Google announced the general availability of its most awaited service Cloud Function at its Google Cloud Next ‘18 conference, San Francisco. Google finally managed to board the serverless bus, that it missed two years ago allowing AWS and Azure to reach their current milestones. This move takes the current cloud platform war to a new level. Google’s Cloud Function now directly competes with Amazon’s Lambda and Microsoft’s Azure Functions. Of late, application development has changed massively, with developers now focusing on application logic instead of infrastructure management, thanks to serverless computing. Developers can now prioritize agility, application quality, and faster deployment with zero server management, auto-scaling traffic management, and integrated security.   Source: Google Cloud website Google’s event driven serverless platform showcases the ability to scale automatically, run codes in response to events, pay while your code runs, and zero server management. Cloud Functions can be used to build: Serverless application backends Developers can quickly build highly available, secure and cost-effective applications as a connective layer of logic is present in Cloud Functions that helps integrate and extend GCP and third-party services. In other words, you can directly call your code from any web, mobile, or backend application or trigger it from GCP services. Real-time data processing Cloud Functions can provide a variety of real-time data processing systems as it responds to events from GCP services such as Stackdriver logging, Cloud Storage, and more. This helps developers to execute their code in response to any change in data. Intelligent applications Developers can leverage Cloud Functions to build intelligent applications with Google Cloud AI integration. One can easily introduce pre-trained machine learning models into the application that can later analyze videos, classify images, convert speech to text, perform NLP (natural language processing) and more. Developers can start making the most of Google Cloud Functions unless they are deploying functions written in Node.js 8 or Python, as these still remain in Beta. In addition to Cloud Functions, Google also announced a preview of serverless containers, a refreshed way of running containers in a fully managed environment. You can read more about this release from Google Cloud release notes. Related Links A serverless online store on AWS could save you money. Build one Learn Azure serverless computing for free – Download a free eBook from Microsoft AWS SAM (AWS Serverless Application Model) is now open source!
Read more
  • 0
  • 0
  • 3682

article-image-automate-tasks-using-azure-powershell-and-azure-cli-tutorial
Gebin George
12 Jul 2018
5 min read
Save for later

Automate tasks using Azure PowerShell and Azure CLI [Tutorial]

Gebin George
12 Jul 2018
5 min read
It is no surprise that we commonly face repetitive and time-consuming tasks. For example, you might want to create multiple storage accounts. You would have to follow the same steps multiple times to get your job done. This is why Microsoft supports its Azure services with multiple ways of automating most of the tasks that can be implemented in Azure. In this Azure Powershell tutorial,  we will learn how to automate redundant tasks on Azure cloud. This article is an excerpt from the book, Hands-On Networking with Azure, written by Mohamed Waly. Azure PowerShell PowerShell is commonly used with most Microsoft products, and Azure is no less important than any of these products. You can use Azure PowerShell cmdlets to manage Azure Networking tasks, however, you should be aware that Microsoft Azure has two types of cmdlets, one for the ASM model, and another for the ARM model. The main difference between cmdlets of the ASM model and the ARM model is, there will be an RM added to the cmdlet of the current portal. For example, if you want to create an ASM virtual network, you would use the following cmdlet: New-AzureVirtualNetwork But for the ARM model, you would use the following: New-AzureRMVirtualNetwork Often, this would be the case. But a few Cmdlets are totally different and some others don't even exist in the ASM model and do exist in the ARM model. By default, you can use Azure PowerShell cmdlets in Windows PowerShell, but you will have to install its module first. Installing the Azure PowerShell module There are two ways of installing the Azure PowerShell module on Windows: Download and install the module from the following link: https://www.microsoft.com/web/downloads/platform.aspx Install the module from PowerShell Gallery Installing the Azure PowerShell module from PowerShell Gallery The following are the required steps to get Azure PowerShell installed: Open PowerShell in an elevated mode. To install the Azure PowerShell module for the current portal run the following cmdlet Install-Module AzureRM. If your PowerShell requires a NuGet provider you will be asked to agree to install it, and you will have to agree for the installation policy modification, as the repository is not available on your environment, as shown in the following screenshot: Creating a virtual network in Azure portal using PowerShell To be able to run your PowerShell cmdlets against Azure successfully, you need to log in first to Azure using the following cmdlet: Login-AzureRMAccount Then, you will be prompted to enter the credentials of your Azure account. Voila! You are logged in and you can run Azure PowerShell cmdlets successfully. To create an Azure VNet, you first need to create the subnets that will be attached to this virtual network. Therefore, let's get started by creating the subnets: $NSubnet = New-AzureRMVirtualNetworkSubnetConfig –Name NSubnet -AddressPrefix 192.168.1.0/24 $GWSubnet = New-AzureRMVirtualNetworkSubnetConfig –Name GatewaySubnet -AddressPrefix 192.168.2.0/27 Now you are ready to create a virtual network by triggering the following cmdlet: New-AzureRMVirtualNetwork -ResourceGroupName PacktPub -Location WestEurope -Name PSVNet -AddressPrefix 192.168.0.0/16 -Subnet $NSubnet,$GWSubnet Congratulations! You have your virtual network up and running with two subnets associated to it, one of them is a gateway subnet. Adding address space to a virtual network using PowerShell To add an address space to a virtual network, you need to retrieve the virtual network first and store it in a variable by running the following cmdlet: $VNet = Get-AzureRMVirtualNetwork -ResourceGroupName PacktPub -Name PSVNet Then, you can add the address space by running the following cmdlet: $VNet.AddressSpace.AddressPrefixes.Add("10.1.0.0/16") Finally, you need to save the changes you have made by running the following cmdlet: Set-AzureRmVirtualNetwork -VirtualNetwork $VNet Azure CLI Azure CLI is an open source, cross-platform that supports implementing all the tasks you can do in Azure portal, with commands. Azure CLI comes in two flavors: Azure CLI 2.0: Which supports only the current Azure portal Azure CLI 1.0: Which supports both portals Throughout this book, we will be using Azure CLI 2.0, so let's get started with its installation. Installing Azure CLI 2.0 Perform the following steps to install Azure CLI 2.0: Download Azure CLI 2.0, from the following link: https://azurecliprod.blob.core.windows.net/msi/azure-cli-2.0.22.msi Once downloaded, you can start the installation: Once you click on Install, it will start to validate your environment to check whether it is compatible with it or not, then it starts the installation: Once the installation completes, you can click on Finish, and you are good to go: Once done, you can open cmd, and write az to access Azure CLI commands: Creating a virtual network using Azure CLI 2.0 To create a virtual network using Azure CLI 2.0, you have to follow these steps: Log in to your Azure account using the following command az login, you have to open the URL that pops up on the CLI, and then enter the following code: To create a new virtual network, you need to run the following command: az network vnet create --name CLIVNet --resource-group PacktPub --location westeurope --address-prefix 192.168.0.0/16 --subnet-name s1 --subnet-prefix 192.168.1.0/24 Adding a gateway subnet to a virtual network using Azure CLI 2.0 To add a gateway subnet to a virtual network, you need to run the following command: az network vnet subnet create --address-prefix 192.168.7.0/27 --name GatewaySubnet --resource-group PacktPub --vnet-name CLIVNet Adding an address space to a virtual network using Azure CLI 2.0 To add an address space to a virtual network, you can run the following command: az network vnet update address-prefixes –add <Add JSON String> Remember that you will need to add a JSON string that describes the address space. To summarize, we learned how to automate cloud tasks using PowerShell and Azure CLI. Check out the book Hands-On Networking with Azure, to learn how to build large-scale, real-world apps using Azure networking solutions. Creating Multitenant Applications in Azure Fine Tune Your Web Application by Profiling and Automation Putting Your Database at the Heart of Azure Solutions
Read more
  • 0
  • 0
  • 10408
article-image-applications-with-aws-services-amazon-dynamodb-amazon-kinesis
Natasha Mathur
05 Jul 2018
17 min read
Save for later

Integrate applications with AWS services: Amazon DynamoDB & Amazon Kinesis [Tutorial]

Natasha Mathur
05 Jul 2018
17 min read
AWS provides hybrid capabilities for networking, storage, database, application development, and management tools for secure and seamless integration. In today's tutorial, we will integrate applications with the two popular AWS services namely Amazon DynamoDB and Amazon Kinesis. Amazon DynamoDB is a fast, fully managed, highly available, and scalable NoSQL database service from AWS. DynamoDB uses key-value and document store data models. Amazon Kinesis is used to collect real-time data to process and analyze it. This article is an excerpt from a book 'Expert AWS Development' written by Atul V. Mistry. By the end of this tutorial, you will know how to integrate applications with the relative AWS services and best practices. Amazon DynamoDB The Amazon DynamoDB service falls under the Database category. It is a fast NoSQL database service from Amazon. It is highly durable as it will replicate data across three distinct geographical facilities in AWS regions. It's great for web, mobile, gaming, and IoT applications. DynamoDB will take care of software patching, hardware provisioning, cluster scaling, setup, configuration, and replication. You can create a database table and store and retrieve any amount and variety of data. It will delete expired data automatically from the table. It will help to reduce the usage storage and cost of storing data which is no longer needed. Amazon DynamoDB Accelerator (DAX) is a highly available, fully managed, and in-memory cache. For millions of requests per second, it reduces the response time from milliseconds to microseconds. DynamoDB is allowed to store up to 400 KB of large text and binary objects. It uses SSD storage to provide high I/O performance. Integrating DynamoDB into an application The following diagram provides a high-level overview of integration between your application and DynamoDB: Please perform the following steps to understand this integration: Your application in your programming language which is using an AWS SDK. DynamoDB can work with one or more programmatic interfaces provided by AWS SDK. From your programming language, AWS SDK will construct an HTTP or HTTPS request with a DynamoDB low-level API. The AWS SDK will send a request to the DynamoDB endpoint. DynamoDB will process the request and send the response back to the AWS SDK. If the request is executed successfully, it will return HTTP 200 (OK) response code. If the request is not successful, it will return HTTP error code and error message. The AWS SDK will process the response and send the result back to the application. The AWS SDK provides three kinds of interfaces to connect with DynamoDB. These interfaces are as follows: Low-level interface Document interface Object persistence (high-level) interface Let's explore all three interfaces. The following diagram is the Movies table, which is created in DynamoDB and used in all our examples: Low-level interface AWS SDK programming languages provide low-level interfaces for DynamoDB. These SDKs provide methods that are similar to low-level DynamoDB API requests. The following example uses the Java language for the low-level interface of AWS SDKs. Here you can use Eclipse IDE for the example. In this Java program, we request getItem from the Movies table, pass the movie name as an attribute, and print the movie release year: Let's create the MovieLowLevelExample file. We have to import a few classes to work with the DynamoDB. AmazonDynamoDBClient is used to create the DynamoDB client instance. AttributeValue is used to construct the data. In AttributeValue, name is datatype and value is data: GetItemRequest is the input of GetItem GetItemResult is the output of GetItem The following code will create the dynamoDB client instance. You have to assign the credentials and region to this instance: Static AmazonDynamoDBClient dynamoDB; In the code, we have created HashMap, passing the value parameter as AttributeValue().withS(). It contains actual data and withS is the attribute of String: String tableName = "Movies"; HashMap<String, AttributeValue> key = new HashMap<String, AttributeValue>(); key.put("name", new AttributeValue().withS("Airplane")); GetItemRequest will create a request object, passing the table name and key as a parameter. It is the input of GetItem: GetItemRequest request = new GetItemRequest() .withTableName(tableName).withKey(key); GetItemResult will create the result object. It is the output of getItem where we are passing request as an input: GetItemResult result = dynamoDB.getItem(request); It will check the getItem null condition. If getItem is not null then create the object for AttributeValue. It will get the year from the result object and create an instance for yearObj. It will print the year value from yearObj: if (result.getItem() != null) { AttributeValue yearObj = result.getItem().get("year"); System.out.println("The movie Released in " + yearObj.getN()); } else { System.out.println("No matching movie was found"); } Document interface This interface enables you to do Create, Read, Update, and Delete (CRUD) operations on tables and indexes. The datatype will be implied with data from this interface and you do not need to specify it. The AWS SDKs for Java, Node.js, JavaScript, and .NET provides support for document interfaces. The following example uses the Java language for the document interface in AWS SDKs. Here you can use the Eclipse IDE for the example. In this Java program, we will create a table object from the Movies table, pass the movie name as attribute, and print the movie release year. We have to import a few classes. DynamoDB is the entry point to use this library in your class. GetItemOutcomeis is used to get items from the DynamoDB table. Table is used to get table details: static AmazonDynamoDB client; The preceding code will create the client instance. You have to assign the credentials and region to this instance: String tableName = "Movies"; DynamoDB docClient = new DynamoDB(client); Table movieTable = docClient.getTable(tableName); DynamoDB will create the instance of docClient by passing the client instance. It is the entry point for the document interface library. This docClient instance will get the table details by passing the tableName and assign it to the movieTable instance: GetItemOutcome outcome = movieTable.getItemOutcome("name","Airplane"); int yearObj = outcome.getItem().getInt("year"); System.out.println("The movie was released in " + yearObj); GetItemOutcome will create an outcome instance from movieTable by passing the name as key and movie name as parameter. It will retrieve the item year from the outcome object and store it into the yearObj object and print it: Object persistence (high-level) interface In the object persistence interface, you will not perform any CRUD operations directly on the data; instead, you have to create objects which represent DynamoDB tables and indexes and perform operations on those objects. It will allow you to write object-centric code and not database-centric code. The AWS SDKs for Java and .NET provide support for the object persistence interface. Let's create a DynamoDBMapper object in AWS SDK for Java. It will represent data in the Movies table. This is the MovieObjectMapper.java class. Here you can use the Eclipse IDE for the example. You need to import a few classes for annotations. DynamoDBAttribute is applied to the getter method. If it will apply to the class field then its getter and setter method must be declared in the same class. The DynamoDBHashKey annotation marks property as the hash key for the modeled class. The DynamoDBTable annotation marks DynamoDB as the table name: @DynamoDBTable(tableName="Movies") It specifies the table name: @DynamoDBHashKey(attributeName="name") public String getName() { return name;} public void setName(String name) {this.name = name;} @DynamoDBAttribute(attributeName = "year") public int getYear() { return year; } public void setYear(int year) { this.year = year; } In the preceding code, DynamoDBHashKey has been defined as the hash key for the name attribute and its getter and setter methods. DynamoDBAttribute specifies the column name and its getter and setter methods. Now create MovieObjectPersistenceExample.java to retrieve the movie year: static AmazonDynamoDB client; The preceding code will create the client instance. You have to assign the credentials and region to this instance. You need to import DynamoDBMapper, which will be used to fetch the year from the Movies table: DynamoDBMapper mapper = new DynamoDBMapper(client); MovieObjectMapper movieObjectMapper = new MovieObjectMapper(); movieObjectMapper.setName("Airplane"); The mapper object will be created from DynamoDBMapper by passing the client. The movieObjectMapper object will be created from the POJO class, which we created earlier. In this object, set the movie name as the parameter: MovieObjectMapper result = mapper.load(movieObjectMapper); if (result != null) { System.out.println("The song was released in "+ result.getYear()); } Create the result object by calling DynamoDBMapper object's load method. If the result is not null then it will print the year from the result's getYear() method. DynamoDB low-level API This API is a protocol-level interface which will convert every HTTP or HTTPS request into the correct format with a valid digital signature. It uses JavaScript Object Notation (JSON) as a transfer protocol. AWS SDK will construct requests on your behalf and it will help you concentrate on the application/business logic. The AWS SDK will send a request in JSON format to DynamoDB and DynamoDB will respond in JSON format back to the AWS SDK API. DynamoDB will not persist data in JSON format. Troubleshooting in Amazon DynamoDB The following are common problems and their solutions: If error logging is not enabled then enable it and check error log messages. Verify whether the DynamoDB table exists or not. Verify the IAM role specified for DynamoDB and its access permissions. AWS SDKs take care of propagating errors to your application for appropriate actions. Like Java programs, you should write a try-catch block to handle the error or exception. If you are not using an AWS SDK then you need to parse the content of low-level responses from DynamoDB. A few exceptions are as follows: AmazonServiceException: Client request sent to DynamoDB but DynamoDB was unable to process it and returned an error response AmazonClientException: Client is unable to get a response or parse the response from service ResourceNotFoundException: Requested table doesn't exist or is in CREATING state Now let's move on to Amazon Kinesis, which will help to collect and process real-time streaming data. Amazon Kinesis The Amazon Kinesis service is under the Analytics product category. This is a fully managed, real-time, highly scalable service. You can easily send data to other AWS services such as Amazon DynamoDB, AmazaonS3, and Amazon Redshift. You can ingest real-time data such as application logs, website clickstream data, IoT data, and social stream data into Amazon Kinesis. You can process and analyze data when it comes and responds immediately instead of waiting to collect all data before the process begins. Now, let's explore an example of using Kinesis streams and Kinesis Firehose using AWS SDK API for Java. Amazon Kinesis streams In this example, we will create the stream if it does not exist and then we will put the records into the stream. Here you can use Eclipse IDE for the example. You need to import a few classes. AmazonKinesis and AmazonKinesisClientBuilder are used to create the Kinesis clients. CreateStreamRequest will help to create the stream. DescribeStreamRequest will describe the stream request. PutRecordRequest will put the request into the stream and PutRecordResult will print the resulting record. ResourceNotFoundException will throw an exception when the stream does not exist. StreamDescription will provide the stream description: Static AmazonKinesis kinesisClient; kinesisClient is the instance of AmazonKinesis. You have to assign the credentials and region to this instance: final String streamName = "MyExampleStream"; final Integer streamSize = 1; DescribeStreamRequest describeStreamRequest = new DescribeStreamRequest().withStreamName(streamName); Here you are creating an instance of describeStreamRequest. For that, you will pass the streamNameas parameter to the withStreamName() method: StreamDescription streamDescription = kinesisClient.describeStream(describeStreamRequest).getStreamDescription(); It will create an instance of streamDescription. You can get information such as the stream name, stream status, and shards from this instance: CreateStreamRequest createStreamRequest = new CreateStreamRequest(); createStreamRequest.setStreamName(streamName); createStreamRequest.setShardCount(streamSize); kinesisClient.createStream(createStreamRequest); The createStreamRequest instance will help to create a stream request. You can set the stream name, shard count, and SDK request timeout. In the createStream method, you will pass the createStreamRequest: long createTime = System.currentTimeMillis(); PutRecordRequest putRecordRequest = new PutRecordRequest(); putRecordRequest.setStreamName(streamName); putRecordRequest.setData(ByteBuffer.wrap(String.format("testData-%d", createTime).getBytes())); putRecordRequest.setPartitionKey(String.format("partitionKey-%d", createTime)); Here we are creating a record request and putting it into the stream. We are setting the data and PartitionKey for the instance. It will create the records: PutRecordResult putRecordResult = kinesisClient.putRecord(putRecordRequest); It will create the record from the putRecord method and pass putRecordRequest as a parameter: System.out.printf("Success : Partition key "%s", ShardID "%s" and SequenceNumber "%s".n", putRecordRequest.getPartitionKey(), putRecordResult.getShardId(), putRecordResult.getSequenceNumber()); It will print the output on the console as follows: Troubleshooting tips for Kinesis streams The following are common problems and their solutions: Unauthorized KMS master key permission error: Without authorized permission on the master key, when a producer or consumer application tries to writes or reads an encrypted stream Provide access permission to an application using Key policies in AWS KMS or IAM policies with AWS KMS Sometimes producer becomes writing slower. Service limits exceeded: Check whether the producer is throwing throughput exceptions from the service, and validate what API operations are being throttled. You can also check Amazon Kinesis Streams limits because of different limits based on the call. If calls are not an issue, check you have selected a partition key that allows distributing put operations evenly across all shards, and that you don't have a particular partition key that's bumping into the service limits when the rest are not. This requires you to measure peak throughput and the number of shards in your stream. Producer optimization: It has either a large producer or small producer. A large producer is running from an EC2 instance or on-premises while a small producer is running from the web client, mobile app, or IoT device. Customers can use different strategies for latency. Kinesis Produce Library or multiple threads are useful while writing for buffer/micro-batch records, PutRecords for multi-record operation, PutRecord for single-record operation. Shard iterator expires unexpectedly: The shard iterator expires because its GetRecord methods have not been called for more than 5 minutes, or you have performed a restart of your consumer application. The shard iterator expires immediately before you use it. This might indicate that the DynamoDB table used by Kinesis does not have enough capacity to store the data. It might happen if you have a large number of shards. Increase the write capacity assigned to the shard table to solve this. Consumer application is reading at a slower rate: The following are common reasons for read throughput being slower than expected: Total reads for multiple consumer applications exceed per-shard limits. In the Kinesis stream, increase the number of shards. Maximum number of GetRecords per call may have been configured with a low limit value. The logic inside the processRecords call may be taking longer for a number of possible reasons; the logic may be CPU-intensive, bottlenecked on synchronization, or I/O blocking. We have covered Amazon Kinesis streams. Now, we will cover Kinesis Firehose. Amazon Kinesis Firehose Amazon Kinesis Firehose is a fully managed, highly available and durable service to load real-time streaming data easily into AWS services such as Amazon S3, Amazon Redshift, or Amazon Elasticsearch. It replicates your data synchronously at three different facilities. It will automatically scale as per throughput data. You can compress your data into different formats and also encrypt it before loading. AWS SDK for Java, Node.js, Python, .NET, and Ruby can be used to send data to a Kinesis Firehose stream using the Kinesis Firehose API. The Kinesis Firehose API provides two operations to send data to the Kinesis Firehose delivery stream: PutRecord: In one call, it will send one record PutRecordBatch: In one call, it will send multiple data records Let's explore an example using PutRecord. In this example, the MyFirehoseStream stream has been created. Here you can use Eclipse IDE for the example. You need to import a few classes such as AmazonKinesisFirehoseClient, which will help to create the client for accessing Firehose. PutRecordRequest and PutRecordResult will help to put the stream record request and its result: private static AmazonKinesisFirehoseClient client; AmazonKinesisFirehoseClient will create the instance firehoseClient. You have to assign the credentials and region to this instance: String data = "My Kinesis Firehose data"; String myFirehoseStream = "MyFirehoseStream"; Record record = new Record(); record.setData(ByteBuffer.wrap(data.getBytes(StandardCharsets.UTF_8))); As mentioned earlier, myFirehoseStream has already been created. A record in the delivery stream is a unit of data. In the setData method, we are passing a data blob. It is base-64 encoded. Before sending a request to the AWS service, Java will perform base-64 encoding on this field. A returned ByteBuffer is mutable. If you change the content of this byte buffer then it will reflect to all objects that have a reference to it. It's always best practice to call ByteBuffer.duplicate() or ByteBuffer.asReadOnlyBuffer() before reading from the buffer or using it. Now you have to mention the name of the delivery stream and the data records you want to create the PutRecordRequest instance: PutRecordRequest putRecordRequest = new PutRecordRequest() .withDeliveryStreamName(myFirehoseStream) .withRecord(record); putRecordRequest.setRecord(record); PutRecordResult putRecordResult = client.putRecord(putRecordRequest); System.out.println("Put Request Record ID: " + putRecordResult.getRecordId()); putRecordResult will write a single record into the delivery stream by passing the putRecordRequest and get the result and print the RecordID: PutRecordBatchRequest putRecordBatchRequest = new PutRecordBatchRequest().withDeliveryStreamName("MyFirehoseStream") .withRecords(getBatchRecords()); You have to mention the name of the delivery stream and the data records you want to create the PutRecordBatchRequest instance. The getBatchRecord method has been created to pass multiple records as mentioned in the next step: JSONObject jsonObject = new JSONObject(); jsonObject.put("userid", "userid_1"); jsonObject.put("password", "password1"); Record record = new Record().withData(ByteBuffer.wrap(jsonObject.toString().getBytes())); records.add(record); In the getBatchRecord method, you will create the jsonObject and put data into this jsonObject . You will pass jsonObject to create the record. These records add to a list of records and return it: PutRecordBatchResult putRecordBatchResult = client.putRecordBatch(putRecordBatchRequest); for(int i=0;i<putRecordBatchResult.getRequestResponses().size();i++){ System.out.println("Put Batch Request Record ID :"+i+": " + putRecordBatchResult.getRequestResponses().get(i).getRecordId()); } putRecordBatchResult will write multiple records into the delivery stream by passing the putRecordBatchRequest, get the result, and print the RecordID. You will see the output like the following screen: Troubleshooting tips for Kinesis Firehose Sometimes data is not delivered at specified destinations. The following are steps to solve common issues while working with Kinesis Firehose: Data not delivered to Amazon S3: If error logging is not enabled then enable it and check error log messages for delivery failure. Verify that the S3 bucket mentioned in the Kinesis Firehose delivery stream exists. Verify whether data transformation with Lambda is enabled, the Lambda function mentioned in your delivery stream exists, and Kinesis Firehose has attempted to invoke the Lambda function. Verify whether the IAM role specified in the delivery stream has given proper access to the S3 bucket and Lambda function or not. Verify your Kinesis Firehose metrics to check whether the data was sent to the Kinesis Firehose delivery stream successfully. Data not delivered to Amazon Redshift/Elasticsearch: For Amazon Redshift and Elasticsearch, verify the points mentioned in Data not delivered to Amazon S3, including the IAM role, configuration, and public access. For CloudWatch and IoT, delivery stream not available as target: Some AWS services can only send messages and events to a Kinesis Firehose delivery stream which is in the same region. Verify that your Kinesis Firehose delivery stream is located in the same region as your other services. We completed implementations, examples, and best practices for Amazon DynamoDB and Amazon Kinesis AWS services using AWS SDK. If you found this post useful, do check out the book 'Expert AWS Development' to learn application integration with other AWS services like Amazon Lambda, Amazon SQS, and Amazon SWF. A serverless online store on AWS could save you money. Build one. Why is AWS the preferred cloud platform for developers working with big data? Verizon chooses Amazon Web Services(AWS) as its preferred cloud provider
Read more
  • 0
  • 1
  • 5442

article-image-amazon-cognito-for-secure-mobile-and-web-user-authentication-tutorial
Natasha Mathur
04 Jul 2018
13 min read
Save for later

Amazon Cognito for secure mobile and web user authentication [Tutorial]

Natasha Mathur
04 Jul 2018
13 min read
Amazon Cognito is a user authentication service that enables user sign-up and sign-in, and access control for mobile and web applications, easily, quickly, and securely. In Amazon Cognito, you can create your user directory, which allows the application to work when the devices are not online. Amazon Cognito supports, to scale, millions of users and authenticates users from social identity providers such as Facebook, Google, Twitter, Amazon, or enterprise identity providers, such as Microsoft Active Directory through SAML, or your own identity provider system. Today, we will discuss the AWS Cognito service for simple and secure user authentication for mobile and web applications. With Amazon Cognito, you can concentrate on developing great application experiences for the user, instead of worrying about developing secure and scalable application solutions for handling the access control permissions of users and synchronization across the devices. Let's explore topics that fall under AWS Cognito and see how it can be used for user authentication from AWS. This article is an excerpt from a book 'Expert AWS Development' written by Atul V. Mistry. Amazon Cognito benefits Amazon Cognito is a fully managed service and it provides User Pools for a secure user directory to scale millions of users; these User Pools are easy to set up. Amazon Cognito User Pools are standards-based identity providers, Amazon Cognito supports many identity and access management standards such as OAuth 2.0, SAML 2.0, OAuth 2.0 and OpenID Connect. Amazon Cognito supports the encryption of data in transit or at rest and multi-factor authentication. With Amazon Cognito, you can control access to the backend resource from the application. You can control the users by defining roles and map different roles for the application, so they can access the application resource for which they are authorized. Amazon Cognito can integrate easily with the sign-up and sign-in for the app because it provides a built-in UI and configuration for different federating identity providers. It provides the facility to customize the UI, as per company branding, in front and center for user interactions. Amazon Cognito is eligible for HIPAA-BAA and is compliant with PCI DSS, SOC 1-3, and ISO 27001. Amazon Cognito features Amazon Cognito provides the following features: Amazon Cognito Identity User Pools Federated Identities Amazon Cognito Sync Data synchronization Today we will discuss User Pools and Federated Identities in detail. Amazon Cognito User Pools Amazon Cognito User Pools helps to create and maintain a directory for users and adds sign-up/sign-in to mobile or web applications. Users can sign in to a User Pool through social or SAML-based identity providers. Enhanced security features such as multi-factor authentication and email/phone number verification can be implemented for your application. With AWS Lambda, you can customize your workflows for Amazon Cognito User Pools such as adding application specific logins for user validation and registration for fraud detection. Getting started with Amazon Cognito User Pools You can create Amazon Cognito User Pools through Amazon Cognito Console, AWS Command Line Interface (CLI), or Amazon Cognito Application Programming Interface (API). Now let's understand all these different ways of creating User Pools. Amazon Cognito User Pool creation from the console Please perform the following steps to create a User Pool from the console. Log in to the AWS Management console and select the Amazon Cognito service. It will show you two options, such as Manage your User Pools and Manage Federated Identities, as shown: Select Manage Your User Pools. It will take you to the Create a user pool screen. You can add the Pool name and create the User Pool. You can create this user pool in two different ways, by selecting: Review defaults: It comes with default settings and if required, you can customize it Step through settings: Step by step, you can customize each setting: When you select Review defaults, you will be taken to the review User Pool configuration screen and then select Create pool. When you will select Step through settings, you will be taken to the Attributes screen to customize it. Let's understand all the screens in brief: Attributes: This gives the option for users to sign in with a username, email address, or phone number. You can select standard attributes for user profiles as well create custom attributes. Policies: You can set the password strength, allow users to sign in themselves, and stipulate days until expire for the newly created account. MFA and verifications: This allows you to enable Multi-Factor Authentication, and configure require verification for emails and phone numbers. You create a new IAM role to set permissions for Amazon Cognito that allows you to send SMS message to users on your behalf. Message customizations: You can customize messages to verify an email address by providing a verification code or link. You can customize user invitation messages for SMS and email but you must include the username and a temporary password. You can customize email addresses from SES-verified identities. Tags: You can add tags for this User Pool by providing tag keys and their values. Devices: This provides settings to remember a user's device. It provides options such as Always, User Opt In, and No. App clients: You can add app clients by giving unique IDs and an optional secret key to access this User Pool. Triggers: You can customize workflows and user experiences by triggering AWS Lambda functions for different events. Reviews: This shows you all the attributes for review. You can edit any attribute on the Reviews screen and then click on Create pool. It will create the User Pool. After creating a new User Pool, navigate to the App clients screen. Enter the App client name as CognitoDemo and click on Create app client: Once this Client App is generated, you can click on the show details to see App client secret: Pool Id, App client id, and App client secret are required to connect any application to Amazon Cognito. Now, we will explore an Amazon Cognito User Pool example to sign up and sign in the user. Amazon Cognito example for Android with mobile SDK In this example, we will perform some tasks such as create a new user, request confirmation code for a new user through email, confirm user, user login, and so on. Create a Cognito User Pool: To create a User Pool with the default configuration, you have to pass parameters to the CognitoUserPool constructor, such as application context, userPoolId, clientId, clientSecret, and cognitoRegion (optional): CognitoUserPool userPool = new CognitoUserPool(context, userPoolId, clientId, clientSecret, cognitoRegion); New user sign-up: Please perform the following steps to sign up new users: Collect information from users such as username, password, given name, phone number, and email address. Now, create the CognitoUserAttributes object and add the user value in a key-value pair to sign up for the user: CognitoUserAttributes userAttributes = new CognitoUserAttributes(); String usernameInput = username.getText().toString(); String userpasswordInput = password.getText().toString(); userAttributes.addAttribute("Name", name.getText().toString()); userAttributes.addAttribute("Email", email.getText().toString()); userAttributes.addAttribute("Phone", phone.getText().toString()); userPool.signUpInBackground(usernameInput, userpasswordInput, userAttributes, null, signUpHandler); To register or sign up a new user, you have to call SignUpHandler. It contains two methods: onSuccess and onFailure. For onSuccess, it will call when it successfully registers a new user. The user needs to confirm the code required to activate the account. You have to pass parameters such as Cognito user, confirm the state of the user, medium and destination of the confirmation code, such as email or phone, and the value for that: SignUpHandler signUpHandler = new SignUpHandler() { @Override public void onSuccess(CognitoUser user, boolean signUpConfirmationState, CognitoUserCodeDeliveryDetails cognitoUserCodeDeliveryDetails) { // Check if the user is already confirmed if (signUpConfirmationState) { showDialogMessage("New User Sign up successful!","Your Username is : "+usernameInput, true); } } @Override public void onFailure(Exception exception) { showDialogMessage("New User Sign up failed.",AppHelper.formatException(exception),false); } }; You can see on the User Pool console that the user has been successfully signed up but not confirmed yet: Confirmation code request: After successfully signing up, the user needs to confirm the code for sign-in. The confirmation code will be sent to the user's email or phone. Sometimes it may automatically confirm the user by triggering a Lambda function. If you selected automatic verification when you created the User Pool, it will send the confirmation code to your email or phone. You can let the user know where they will get the confirmation code from the cognitoUserCodeDeliveryDetails object. It will indicate where you will send the confirmation code: VerificationHandler resendConfCodeHandler = new VerificationHandler() { @Override public void onSuccess(CognitoUserCodeDeliveryDetails details) { showDialogMessage("Confirmation code sent.","Code sent to "+details.getDestination()+" via "+details.getDeliveryMedium()+".", false); } @Override public void onFailure(Exception exception) { showDialogMessage("Confirmation code request has failed", AppHelper.formatException(exception), false); } }; In this case, the user will receive an email with the confirmation code: The user can complete the sign-up process after entering the valid confirmation code. To confirm the user, you need to call the GenericHandler. AWS SDK uses this GenericHandler to communicate the result of the confirmation API: GenericHandler confHandler = new GenericHandler() { @Override public void onSuccess() { showDialogMessage("Success!",userName+" has been confirmed!", true); } @Override public void onFailure(Exception exception) { showDialogMessage("Confirmation failed", exception, false); } }; Once the user confirms, it will be updated in the Amazon Cognito console: Sign in user to the app: You must create an authentication callback handler for the user to sign in to your application. The following code will show you how the interaction happens from your app and SDK: // call Authentication Handler for User sign-in process. AuthenticationHandler authHandler = new AuthenticationHandler() { @Override public void onSuccess(CognitoUserSession cognitoUserSession) { launchUser(); // call Authentication Handler for User sign-in process. AuthenticationHandler authHandler = new AuthenticationHandler() { @Override public void onSuccess(CognitoUserSession cognitoUserSession) { launchUser(); } @Override public void getAuthenticationDetails(AuthenticationContinuation continuation, String username) { // Get user sign-in credential information from API. AuthenticationDetails authDetails = new AuthenticationDetails(username, password, null); // Send this user sign-in information for continuation continuation.setAuthenticationDetails(authDetails); // Allow user sign-in process to continue continuation.continueTask(); } @Override public void getMFACode(MultiFactorAuthenticationContinuation mfaContinuation) { // Get Multi-factor authentication code from user to sign-in mfaContinuation.setMfaCode(mfaVerificationCode); // Allow user sign-in process to continue mfaContinuation.continueTask(); } @Override public void onFailure(Exception e) { // User Sign-in failed. Please check the exception showDialogMessage("Sign-in failed", e); } @Override public void authenticationChallenge(ChallengeContinuation continuation) { /** You can implement Custom authentication challenge logic * here. Pass the user's responses to the continuation. */ } }; Access AWS resources from application user: A user can access AWS resource from the application by creating an AWS Cognito Federated Identity Pool and associating an existing User Pool with that Identity Pool, by specifying User Pool ID and App client id. Please see the next section (Step 5) to create the Federated Identity Pool with Cognito. Let's continue with the same application; after the user is authenticated, add the user's identity token to the logins map in the credential provider. The provider name depends on the Amazon Cognito User Pool ID and it should have the following structure: cognito-idp.<USER_POOL_REGION>.amazonaws.com/<USER_POOL_ID> For this example, it will be: cognito-idp.us-east-1.amazonaws.com/us-east-1_XUGRPHAWA. Now, in your credential provider, pass the ID token that you get after successful authentication: // After successful authentication get id token from // CognitoUserSession String idToken = cognitoUserSession.getIdToken().getJWTToken(); // Use an existing credential provider or create new CognitoCachingCredentialsProvider credentialsProvider = new CognitoCachingCredentialsProvider(context, IDENTITY_POOL_ID, REGION); // Credentials provider setup Map<String, String> logins = new HashMap<String, String>(); logins.put("cognito-idp.us-east-1.amazonaws.com/us-east-1_ XUGRPHAWA", idToken); credentialsProvider.setLogins(logins); You can use this credential provider to access AWS services, such as Amazon DynamoDB, as follows: AmazonDynamoDBClient dynamoDBClient = new AmazonDynamoDBClient(credentialsProvider) You have to provide the specific IAM permission to access AWS services, such as DynamoDB. You can add this permission to the Federated Identities, as mentioned in the following Step 6, by editing the View Policy Document. Once you have attached the appropriate policy, for example, AmazonDynamoDBFullAccess, for this application, you can perform the operations such as create, read, update, and delete operations in DynamoDB. Now, we will look at how to create the Amazon Cognito Federated Identities. Amazon Cognito Federated Identities Amazon Cognito Federated Identities enables you to create unique identities for the user and, authenticate with Federated Identity Providers. With this identity, the user will get temporary, limited-privilege AWS credentials. With these credentials, the user can synchronize their data with Amazon Cognito Sync or securely access other AWS services such as Amazon S3, Amazon DynamoDB, and Amazon API Gateway. It supports Federated Identity providers such as Twitter, Amazon, Facebook, Google, OpenID Connect providers, or SAML identity providers, unauthenticated identities. It also supports developer-authenticated identities from which you can register and authenticate the users through your own backend authentication systems. You need to create an Identity Pool to use Amazon Cognito Federated Identities in your application. This Identity Pool is specific for your account to store user identity data. Creating a new Identity Pool from the console Please perform the following steps to create a new Identity Pool from the console: Log in to the AWS Management console and select the Amazon Cognito Service. It will show you two options: Manage your User Pools and Manage Federated Identities. Select Manage Federated Identities. It will navigate you to the Create new identity pool screen. Enter a unique name for the Identity pool name: You can enable unauthenticated identities by selecting Enable access to unauthenticated identities from the collapsible section: Under Authentication providers, you can allow your users to authenticate using any of the authentication methods. Click on Create pool. You must select at least one identity from Authentication providers to create a valid Identity Pool. Here Cognito has been selected for a valid Authentication provider by adding User Pool ID and App client id: It will navigate to the next screen to create a new IAM role by default, to provide limited permission to end users. These permissions are for Cognito Sync and Mobile Analytics but you can edit policy documents to add/update permissions for more services. It will create two IAM roles. One for authenticated users that are supported by identity providers and another for unauthenticated users, known as guest users. Click Allow to generate the Identity Pool: Once the Identity Pool is generated, it will navigate to the Getting started with Amazon Cognito screen for that Identity Pool. Here, it will provide you with downloadable AWS SDK for different platforms such as Android, iOS - Objective C, iOS - Swift, JavaScript, Unity, Xamarin, and .NET. It also provides sample code for Get AWS Credentials and Store User Data: You have created Amazon Cognito Federated Identities. We looked at how user authentication process in AWS Cognito works with User Pools and Federated Identities. If you found this post useful, check out the book 'Expert AWS Development' to learn other concepts such as Amazon Cognito sync, traditional web hosting etc, in AWS development. Keep your serverless AWS applications secure [Tutorial] Amazon Neptune, AWS’ cloud graph database, is now generally available How to start using AWS
Read more
  • 0
  • 3
  • 8321