Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Newsletter Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

Tech News

3711 Articles
article-image-startup-focus-sea-machines-winning-contracts-for-autonomous-marine-systems-from-ai-trends
Matthew Emerick
15 Oct 2020
8 min read
Save for later

Startup Focus: Sea Machines Winning Contracts for Autonomous Marine Systems from AI Trends

Matthew Emerick
15 Oct 2020
8 min read
By AI Trends Staff The ability to add automation to an existing marine vessel to make it autonomous is here today and is being proven by a Boston company. Sea Machines builds autonomous vessel software and systems for the marine industry. Founded in 2015, the company recently raised $15 million in a Series B round, making it total raised $27.5 million since 2017.  Founder and CEO Michael G. Johnson, a licensed marine engineer, recently took the time to answer via email some questions AI Trends poses to selected startups. Describe your team, the key people Sea Machines is led by a team of mariners, engineers, coders and autonomy scientists. The company today has a crew of 30 people based in Boston; Hamburg, Germany; and Esbjerg, Denmark. Sea Machines is also hiring for a variety of positions, which can be viewed at sea-machines.com/careers. Michael Johnson, Founder and CEO, Sea Machines What business problem are you trying to solve? The global maritime industry is responsible for billions in economic output and is a major driver of jobs and commerce. Despite the sector’s success and endurance, it faces significant challenges that can negatively impact operator safety, performance and profitability. Sea Machines is solving many of these challenges by developing technologies that are helping the marine industry transition into a new era of task-driven, computer-guided vessel operations.   How does your solution address the problem? Autonomous systems solve for these challenges in several ways: Autonomous grid and waypoint following capabilities relieve mariners from manually executing planned routes. Today’s autonomous systems uniquely execute with human-like behavior, intelligently factoring in environmental and sea conditions (including wave height, pitch, heave and roll); change speeds between waypoints; and actively detect obstacles for collision avoidance purposes. Autonomous marine systems also enable optionally manned or autonomous-assist (reduced crew) modes that can reduce mission delays and maximize effort. This is an important feature for anyone performing time-sensitive operations, such as on-water search-and-rescues or other urgent missions. Autonomous marine systems offer obstacle detection and collision avoidance capabilities that keep people and assets safe and out of harm’s way. These advanced technologies are much more reliable and accurate than the human eye, especially in times of low light or in poor sea conditions. Because today’s systems enable remote-helm control and remote payload management, there is a reduced need for mariners (such as marine fire or spill response crews) to physically man a vessel in a dangerous environment. A remote-helm control beltpack also improves visibility by enabling mariners to step outside of the wheelhouse to whatever location provides the best vantage point when performing tight maneuvers, dockings and other precision operations. Autonomous marine systems enable situational awareness with multiple cameras and sensors streaming live over a 4G connection. This real-time data allows shoreside or at-sea operators a full view of an autonomous vessel’s environment, threats and opportunities. Minimally manned vessels can autonomously collaborate to cover more ground with less resources required, creating a force-multiplier effect. A single shoreside operator can command multiple autonomous boats with full situational awareness. These areas of value overlap for all sectors but for the government and military sector, new on-water capabilities and unmanned vessels are a leading driver. By contrast, the commercial sector is looking for increased productivity, efficiency, and predictable operations. Our systems meet all of these needs. Our technology is designed to be installed on new vessels as well as existing vessels. Sea Machines’ ability to upgrade existing fleets greatly reduces the time and cost to leverage the value of our autonomous systems.  How are you getting to the market? Is there competition? Sea Machines has an established dealer program to support the company’s global sales across key commercial marine markets. The program includes many strategic partners who are enabled to sell, install and service the company’s line of intelligent command and control systems for workboats. To date, Sea Machines dealers are located across the US and Canada, in Europe, in Singapore and UAE. We have competition for autonomous marine systems, but our products are the only ones that are retrofit ready, not requiring new vessels to be built. Do you have any users or customers?   Yes we have achieved significant sales traction since launching our SM series of products in 2018.  Just since the summer, Sea Machines has been awarded several significant contracts and partnerships:  The first allowed us to begin serving the survey vessel market with the first announced collaboration with DEEP BV in the Netherlands. DEEP’s vessel outfitted with the SM300 entered survey service very recently.  Next, we partnered with Castine-based Maine Maritime Academy (MMA) and representatives of the U.S. Maritime Administration (MARAD)’s Maritime Environmental and Technical Assistance (META) Program to bring valuable, hands-on education about autonomous marine systems into the MMA curriculum.  Then we recently announced a partnership with shipbuilder Metal Shark Boats, of Jeanerette, Louisiana, to supply the U.S. Coast Guard (USCG)’s Research and Development Center (RDC) with a new Sharktech 29 Defiant vessel for the purposes of testing and evaluating the capabilities of available autonomous vessel technology. USCG demonstrations are happening now (through November 5) off the coast of Hawaii.  Finally, just this month, we announced that the U.S. Department of Defense (DOD)’s Defense Innovation Unit (DIU) awarded us with a multi-year Other Transaction (OT) agreement. The primary purpose of the agreement is to initiate a prototype that will enable commercial ocean-service barges as autonomous Forward Arming and Refueling Point (FARP) units for an Amphibious Maritime Projection Platform (AMPP). Specifically, Sea Machines will engineer, build and demonstrate ready-to-deploy system kits that enable autonomous, self-propelled operation of opportunistically available barges to land and replenish military aircraft. In the second half of 2020 we are also commencing onboard collaborations with some crew-transfer vessel (CTV) operators serving the wind farm industry. How is the company funded? The company recently completed a successful Series B round, which provided $15M in funds, with a total amount raised of $27.5M since 2017. The most recent funds we were able to raise are going to significantly impact Sea Machines, and therefore the maritime and marine industries as a whole. The funds will be put to use to further strengthen our technical development team as well as build out our next level of systems manufacturing and scale our operations group to support customer deployments.  We will also be investing in some supporting technologies to speed our course to full dock-to-dock, over-the-horizon autonomy. The purpose of our technology is to optimize vessel operations with increased performance, productivity, predictability and ultimately safety. In closing, we’d like to add that the marine industries are a critically significant component of the global economy and it’s up to us to keep it strong and relevant. Along with people, processes and capital, pressing the bounds of technology is a key driver. The world is being revolutionized by intelligent and autonomous self-piloting technology and today we find ourselves just beyond the starting line of a busy road to broad adoption through all marine sectors. If Sea Machines continues to chart the course with forward-looking pertinence, then you will see us rise up to become one of the most significant companies and brands serving the industry in the 21st century.   Any anecdotes/stories? This month we released software version 1.7 on our SM300. That’s seven significant updates in just over 18 months, each one providing increased technical hardening and new features for specific workboat sectors.  Another interesting story is about our Series B funding, which, due to the pandemic, we raised virtually.  Because of where we are as a company, we have been proving our ability to retool the marine industry with our technology, and therefore we are delivering confidence to investors. We were forced to conduct the entire process by video conference, which may have increased overall efficiency of the raise as these rounds traditionally require thousands if not tens of thousands of miles of travel for face-to-face meetings, diligence, and handshakes. Remote pitches also proved to be an advantage because it allowed us to showcase our technology in a more direct way. We did online demos where we had our team remotely connected to our vessels off Boston Harbor. We were able to get the investors into the captain’s chair, as if they were remotely commanding a vessel in real-world operations. Finally, in January, we announced the receipt of ABS and USCG approval for our SM200 wireless helm and control systems on a major class of U.S.-flag articulated tug-barges (ATBs), the first unit has been installed and is in operation, and we look forward to announcing details around it.  We will be taking the SM200 forward into the type-approval process. Learn more at Sea Machines.
Read more
  • 0
  • 0
  • 3293

article-image-web-applications-are-focus-of-cybercrime-gangs-in-data-breaches-report-finds-from-ai-trends
Matthew Emerick
15 Oct 2020
7 min read
Save for later

Web Applications are Focus of Cybercrime Gangs in Data Breaches, Report Finds from AI Trends

Matthew Emerick
15 Oct 2020
7 min read
By John P. Desmond, AI Trends Editor Web applications are the primary focus of many cybercrime gangs engaged in data breaches, a primary security concern to retailers, according to the 2020 Data Breach Investigations Report (DBIR) recently released by Verizon, in its 13th edition of the report. Verizon analyzed a total of 157,525 incidents; 3,950 were confirmed data breaches.  “These data breaches are the most serious type of incident retailers face. Such breaches generally result in the loss of customer data, including, in the worst cases, payment data and log-in and password combinations,” stated Ido Safruti, co-founder and chief technology officer, PerimeterX, a provider of security services for websites, in an account in Digital Commerce 360. Among the reports highlights: Misconfiguration errors, resulting from failure to implement all security controls, top the list of the fastest-growing risk to web applications. Across all industries, misconfiguration errors increased from below 20 percent in the 2017 survey to over 40 percent in the 2020 survey. “The reason for this is simple,” Safruti stated. “Web applications are growing more and more complex. What were formerly websites are now full-blown applications made up of dozens of components and leveraging multiple external services.” Ido Safruti, co-founder and chief technology officer, PerimeterX External code can typically comprise 70 percent or more of web applications, many of them JavaScript calls to external libraries and services. “A misconfigured service or setting for any piece of a web application offers a path to compromise the application and skim sensitive customer data,” Safruti stated. Cybercriminal gangs work to exploit rapid changes on web applications, as development teams build and ship new code faster and faster, often tapping third-party libraries and services. Weaknesses in version control and monitoring of changes to web applications for unauthorized introductions of code, are vulnerabilities. Magecart attacks, from a consortium of malicious hacker groups who target online shopping cart systems especially on large ecommerce sites, insert rogue elements as components of Web applications with the goal of stealing credit card data of shoppers.  “Retailers should consider advanced technology using automated and audited processes to manage configuration changes,” Safruti advises. Vulnerabilities are not patched quickly enough, leaving holes for attacks to exploit. Only half of vulnerabilities are patched within three months of discovery, the 2020 DBIR report found. These attacks offer hackers the potential of  large amounts of valuable customer information with the least amount of effort.   Attacks against web application servers made up nearly 75% of breached assets in 2019, up from roughly 50% in 2017, the DBIR report found. Organized crime groups undertook roughly two-thirds of breaches and 86% of breaches were financially motivated. The global average cost of a data breach is $3.92 million, with an average of over $8 million in the United States, according to a 2019 study from the Ponemon Institute, a research center focused on privacy, data protection and information security. Another analysis of the 2020 DBIT report found that hacking and social attacks have leapfrogged malware as the top attack tactic. “Sophisticated malware is no longer necessary to perform an attack,” stated the report in SecurityBoulevard.  Developers and QA engineers who develop and test web applications would benefit from the use of automated security testing tools and security processes that integrate with their workflow. “We believe developers and DevOps personnel are one of the weakest links in the chain and would benefit the most from remediation techniques,” the authors stated. Credential Stuffing Attack Exploit Users with Same Password Across Sites Credential stuffing is a cyberattack where lists of stolen usernames and/or email addresses are used to gain unauthorized access to user accounts through large-scale automated login requests directed against a web application.  “Threat actors are always conducting credential stuffing attacks,” found a “deep dive” analysis of the 2020 DBIR report from SpyCloud, a security firm focused on preventing online fraud.   The SpyCloud researchers advise users never to reuse passwords across online accounts. “Password reuse is a major factor in credential stuffing attacks,” the authors state. They advise using a password manager and storing a unique complex password for each account. The 2020 DBIR report found this year’s top malware variant to be password dumpers, malware that extracts passwords from infected systems. This malware is aimed at acquiring credentials stored on target computers, or involve keyloggers that acquire credentials as users enter them.  Some 22 percent of breaches found were the result of social attacks, which are cyber attacks that involve social engineering and phishing. Phishing – making fake websites, emails, text messages, and social media messages to impersonate trusted entities – is still a major way that sensitive authentication credentials are acquired illicitly, SpyCloud researchers found. Average consumers are each paying more than $290 in out-of-pocket costs and spending 16 hours to resolve the effects of this data loss and the resultant account takeover, SpyCloud found.  Business Increasing Investment in AI for Cybersecurity, Capgemini Finds To defend against the new generation of cyberattacks, businesses are increasing their investment in AI systems to help. Two-thirds of organizations surveyed by Capgemini Research last year said they will not be able to respond to critical threats without AI. Capgemini surveyed 850 senior IT executives from IT information security, cybersecurity and IT operations across 10 countries and seven business sectors. Among the highlights was that AI-enabled cybersecurity is now an imperative: Over half (56%) of executives say their cybersecurity analysts are overwhelmed by the vast array of data points they need to monitor to detect and prevent intrusion. In addition, the type of cyberattacks that require immediate intervention, or that cannot be remediated quickly enough by cyber analysts, have notably increased, including: cyberattacks affecting time-sensitive applications (42% saying they had gone up, by an average of 16%). automated, machine-speed attacks that mutate at a pace that cannot be neutralized through traditional response systems (43% reported an increase, by an average of 15%). Executives interviewed cited benefits of using AI in cybersecurity:  64% said it lowers the cost of detecting breaches and responding to them – by an average of 12%. 74% said it enables a faster response time: reducing time taken to detect threats, remedy breaches and implement patches by 12%. 69% also said AI improves the accuracy of detecting breaches, and 60% said it increases the efficiency of cybersecurity analysts, reducing the time they spend analyzing false positives and improving productivity. Budgets for AI in cybersecurity are projected to rise, with almost half (48%) of respondents said they are planning 29 percent increases in FY2020; some 73 percent were testing uses cases for AI in cybersecurity; only one in five organizations reported using AI in cybersecurity before 2019. “AI offers huge opportunities for cybersecurity,” stated Oliver Scherer, CISO of Europe’s leading consumer electronics retailer, MediaMarktSaturn Retail Group, in the Capgemini report. “This is because you move from detection, manual reaction and remediation towards an automated remediation, which organizations would like to achieve in the next three or five years.” Geert van der Linden, Cybersecurity Business Lead, Capgemini Group Barriers remain, including a lack of understanding in how to scale use cases from proof of concept to full-scale deployment.   “Organizations are facing an unparalleled volume and complexity of cyber threats and have woken up to the importance of AI as the first line of defense,” stated Geert van der Linden, Cybersecurity Business Lead at Capgemini Group. “As cybersecurity analysts are overwhelmed, close to a quarter of them declaring they are not able to successfully investigate all identified incidents, it is critical for organizations to increase investment and focus on the business benefits that AI can bring in terms of bolstering their cybersecurity.” Read the source articles in the 2020 Data Breach Investigations Report from Verizon,  in Digital Commerce 360, in SecurityBoulevard, from SpyCloud and from Capgemini Research.  
Read more
  • 0
  • 0
  • 3143

article-image-the-best-google-cloud-courses-to-future-proof-your-resume-from-android-development-android-authority
Matthew Emerick
15 Oct 2020
4 min read
Save for later

The best Google Cloud courses to future-proof your resume from Android Development – Android Authority

Matthew Emerick
15 Oct 2020
4 min read
Google Cloud certification demonstrates proficiency using the Google Cloud Platform (GCP). The best Google Cloud courses prepare you for these exams so you can add the certification to your resume quickly and on your first try. The Google Cloud Platform is a suite of tools hosted on the cloud that businesses can use to enhance their own services. These include cloud storage, security, user profiles, machine learning, database management, IoT, and more. With Google Cloud certification, IT professionals will open doors in their careers and potentially command a higher salary. See also:  What is Google Cloud certification and should I get it? Gaining certification is simple. Google offers a total of seven certificates. Each represents an understanding of specific tools offered by the platform as they apply to respective careers. Before sitting any of these exams, it’s important that professionals gain sufficient knowledge of the technologies they are being tested on. This will ensure a passing grade and avoid resits. Below, you will find some of the best Google Cloud certification courses that will prepare you for these exams. Editor’s note: We’ll be adding to this list over time as more Google Cloud courses become available. The best Google Cloud courses The Complete Google Cloud Mastery Bundle With seven different certifications and a huge roster of different skills and tools, it can be tough to know where to start with your Google Cloud certification. That’s what makes this Complete Google Cloud Mastery Bundle such a great option. It provides eight separate learning kits which include specific training relating to each of the exams, as well as a separate exam preparation “boot camp.” The course would normally cost $1,392, but Android Authority readers can get the full bundle for just $39. This really is one of the best Google Cloud courses you can find right now, at an excellent price. Why you should buy Includes an exam preparation boot camp Amazing value Information for every type of professional Why you should pass Some information may not be relevant for certain professionals Exam Bootcamp is for 2019 $33 .15 The Complete Google Cloud Mastery Bundle Use offer code: MERRYSAVE15 Save $1358 .85 Buy it Now The Complete Google Cloud Mastery Bundle Buy it Now Save $1358 .85 $33 .15 Use offer code: MERRYSAVE15 The Complete 2020 Cloud Foundation Certification Bundle Credit: Adam Sinicki / Android Authority We’ve previously discussed the benefits of Google Cloud Platform certification versus Microsoft Azure and Amazon Web Services, but ultimately the best option is to get them all. If you’re unsure of where to start, then this introductory course that teaches the basics of all three is perfect. You’ll get lifetime access to four separate courses containing 160 lessons. By the end, you’ll be ready to sit multiple exams (including CompTIA Cloud+). Why you should buy Perfect for a well-rounded knowledge Prepares you for multiple exams A large amount of content Why you should pass Not necessary if you only want Google Cloud certification $49 .99 The Complete 2020 Cloud Foundation Certification Bundle Save $1130 .01 Buy it Now The Complete 2020 Cloud Foundation Certification Bundle Buy it Now Save $1130 .01 $49 .99 The Google Cloud Certifications Practice Tests + Courses Bundle Credit: Adam Sinicki / Android Authority This is another extremely comprehensive option that includes training for all seven tests as well as practice questions. Practice tests are a great thing to look for when choosing Google Cloud courses, as they can build confidence and ensure you are ready. This course also represents fantastic value at just $29.99 (down from $639). Why you should buy Preparation for all seven exams Includes practice questions Why you should pass May have more content than you need $49 .99 The Complete 2020 Cloud Foundation Certification Bundle Save $1130 .01 Buy it Now The Complete 2020 Cloud Foundation Certification Bundle Buy it Now Save $1130 .01 $49 .99 GCP: Complete Google Data Engineer and Cloud Architect Guide This course provides access to 166 lectures and 22 hours of content, all squarely aimed at the Data Engineer and Cloud Architect certifications. This focus makes the course a great choice for anyone that knows these are their areas of interest. It also gives you a large amount of highly focussed content, which will prepare you for either the Google Data Engineer or Cloud Architect certification exams. Once again, Android Authority readers can get a large discount. The course is usually valued at $199, but if you act now, you can get the whole thing for just $15. Why you should buy A large amount of focussed content Data science and cloud technologies are up-and-coming areas Specific exam preparation Why you should pass Only relevant for the two mentioned certificates $9 .00 GCP: Complete Google Data Engineer and Cloud Architect Guide Use offer code: SPRINGSAVE40 Save $190 .00 Buy it Now GCP: Complete Google Data Engineer and Cloud Architect Guide Buy it Now Save $190 .00 $9 .00 Use offer code: SPRINGSAVE40  
Read more
  • 0
  • 0
  • 3214

article-image-ai-autonomous-cars-might-have-just-a-four-year-endurance-lifecycle-from-ai-trends
Matthew Emerick
15 Oct 2020
14 min read
Save for later

AI Autonomous Cars Might Have Just A Four-Year Endurance Lifecycle from AI Trends

Matthew Emerick
15 Oct 2020
14 min read
By Lance Eliot, the AI Trends Insider   After AI autonomous self-driving cars have been abundantly fielded onto our roadways, one intriguing question that has so far gotten scant attention is how long will those self-driving cars last.    It is easy to simply assume that the endurance of a self-driving car is presumably going to be the same as today’s conventional cars, especially since most of the self-driving cars are currently making use of a conventional car rather than a special-purpose built vehicle.  But there is something to keep in mind about self-driving cars that perhaps does not immediately meet the eye, namely, they are likely to get a lot of miles in a short period. Given that the AI is doing the driving, there is no longer a dampening on the number of miles that a car might be driven in any noted time period, which usually is based on the availability of a human driver. Instead, the AI is a 24 x 7 driver that can be used non-stop and attempts to leverage the self-driving car into a continuously moving and available ride-sharing vehicle.  With all that mileage, the number of years of endurance is going to be lessened in comparison to a comparable conventional car that is driven only intermittently. You could say that the car is still the car, while the difference is that the car might get as many miles of use in a much shorter period of time and thus reach its end-of-life sooner (though nonetheless still racking up the same total number of miles).  Some automotive makers have speculated that self-driving cars might only last about four years.  This comes as quite a shocking revelation that AI-based autonomous cars might merely be usable for a scant four years at a time and then presumably end-up on the scrap heap.  Let’s unpack the matter and explore the ramifications of a presumed four-year life span for self-driving cars.  For my framework about AI autonomous cars, see the link here: https://aitrends.com/ai-insider/framework-ai-self-driving-driverless-cars-big-picture/  Why this is a moonshot effort, see my explanation here: https://aitrends.com/ai-insider/self-driving-car-mother-ai-projects-moonshot/  For more about the levels as a type of Richter scale, see my discussion here: https://aitrends.com/ai-insider/richter-scale-levels-self-driving-cars/  For the argument about bifurcating the levels, see my explanation here: https://aitrends.com/ai-insider/reframing-ai-levels-for-self-driving-cars-bifurcation-of-autonomy/  Life Span Of Cars  According to various stats about today’s cars, the average age of a conventional car in the United States is estimated at 11.6 years old.  Some tend to use the 11.6 years or a rounded 12 years as a surrogate for how long a car lasts in the U.S, though this is somewhat problematic to do since the average age is not the endpoint of a car and encapsulates a range of ages of cars, including a slew of cars that were retired at a much younger age and those that hang-on to a much older age.  Indeed, one of the fastest-growing segments of car ages is the group that is 16 years or older, amounting to an estimated 81 million such cars by the year 2021. Of those 81 million cars, around one-fourth are going to be more than 25 years old.  In short, cars are being kept around longer and longer.  When you buy a new car, the rule-of-thumb often quoted by automakers is that the car should last about 8 years or 150,000 miles.  This is obviously a low-ball kind of posturing, trying to set expectations so that car buyers will be pleased if their cars last longer. One supposes it also perhaps gets buyers into the mental mode of considering buying their next car in about eight years or so.  Continuing the effort to consider various stats about cars, Americans drive their cars for about 11,000 miles per year. If a new car is supposed to last for 150,000 miles, the math then suggests that at 11,000 miles per year you could drive the car for 14 years (that’s 150,000 miles divided by 11,000 miles per year).  Of course, the average everyday driver is using their car for easy driving such as commuting to work and driving to the grocery store. Generally, you wouldn’t expect the average driver to be putting many miles onto a car.  What about those that are pushing their cars to the limit and driving their cars in a much harsher manner?  Various published stats about ridesharing drivers such as Uber and Lyft suggest that they are amassing about 1,000 miles per week on their cars. If so, you could suggest that the number of miles per year would be approximately 50,000 miles. At the pace of 50,000 miles per year, presumably, these on-the-go cars would only last about 3 years, based on the math of 150,000 miles divided by 50,000 miles per year.  In theory, this implies that a ridesharing car being used today will perhaps last about 3 years.  For self-driving cars, most would agree that a driverless car is going to be used in a similar ridesharing manner and be on-the-road quite a lot.  This seems sensible. To make as much money as possible with a driverless car, you would likely seek to maximize the use of it. Put it onto a ridesharing network and let it be used as much as people are willing to book it and pay to use it.  Without the cost and hassle of having to find and use a human driver for a driverless car, the AI will presumably be willing to drive a car whenever and however long is needed. As such, a true self-driving car is being touted as likely to be running 24×7.  In reality, you can’t actually have a self-driving car that is always roaming around, since there needs to be time set aside for ongoing maintenance of the car, along with repairs, and some amount of time for fueling or recharging of the driverless car.  Overall, it would seem logical to postulate that a self-driving car will be used at least as much as today’s human-driven ridesharing cars, plus a lot more so since the self-driving car is not limited by human driving constraints.  In short, if it is the case that today’s ridesharing cars are hitting their boundaries at perhaps three to five years, you could reasonably extend that same thinking to driverless cars and assume therefore that self-driving cars might only last about four years.  The shock that a driverless car might only last four years is not quite as surprising when you consider that a true self-driving car is going to be pushed to its limits in terms of usage and be a ridesharing goldmine (presumably) that will undergo nearly continual driving time.  For why remote piloting or operating of self-driving cars is generally eschewed, see my explanation here: https://aitrends.com/ai-insider/remote-piloting-is-a-self-driving-car-crutch/  To be wary of fake news about self-driving cars, see my tips here: https://aitrends.com/ai-insider/ai-fake-news-about-self-driving-cars/  The ethical implications of AI driving systems are significant, see my indication here: http://aitrends.com/selfdrivingcars/ethically-ambiguous-self-driving-cars/  Be aware of the pitfalls of normalization of deviance when it comes to self-driving cars, here’s my call to arms: https://aitrends.com/ai-insider/normalization-of-deviance-endangers-ai-self-driving-cars/  Factors Of Car Aging  Three key factors determine how long a car will last, namely:  How the car was built  How the car is used  How the car is maintained  Let’s consider how those key factors apply to self-driving cars.  In the case of today’s early versions of what are intended to be driverless cars, by-and-large most of the automakers are using a conventional car as the basis for their driverless car, rather than building an entirely new kind of car.  We will eventually see entirely new kinds of cars being made to fully leverage a driverless car capability, but for right now it is easier and more expedient to use a conventional car as the cornerstone for an autonomous car.  Therefore, for the foreseeable future, we can assume that the manner of how a driverless car was built is in keeping with how a conventional car is built, implying that the car itself will last as long as a conventional car might last.  In terms of car usage, as already mentioned, a driverless car is going to get a lot more usage than the amount of driving by an average everyday driver and be used at least as much as today’s ridesharing efforts. The usage is bound to be much higher.  The ongoing maintenance of a self-driving car will become vital to the owner of a driverless car.  I say this because any shortcomings in the maintenance would tend to mean that the driverless car will be in the shop and not be as available on the streets. The revenue stream from an always-on self-driving car will be a compelling reason for owners to make sure that their self-driving car is getting the proper amount of maintenance.  In that sense, the odds would seem to be the case that a driverless car will likely be better maintained than either an average everyday car or even today’s ridesharing cars.  One additional element to consider for driverless cars consists of the add-ons for the sensory capabilities and the computer processing aspects. Those sensory devices such as cameras, radar, ultrasonic, LIDAR, and so on, need to be factored into the longevity of the overall car, and the same applies to the computer chips and memory on-board too.  Why Retire A Car  The decision to retire a car is based on a trade-off between trying to continue to pour money into a car that is breaking down and excessively costing money to keep afloat, versus ditching the car and opting to get a new or newer car instead.  Thus, when you look at how long a car will last, you are also silently considering the cost of a new or newer car.  We don’t yet know what the cost of a driverless car is going to be.  If the cost is really high to purchase a self-driving car, you would presumably have a greater incentive to try and keep a used self-driving car in sufficient working order.  There is also a safety element that comes to play in deciding whether to retire a self-driving car.  Suppose a driverless car that is being routinely maintained is as safe as a new self-driving car, but eventually, the maintenance can only achieve so much in terms of ensuring that the driverless car remains as safe while driving on the roadways as would be a new or newer self-driving car.  The owner of the used self-driving car would need to ascertain whether the safety degradation means that the used driverless car needs to be retired.  Used Market For Self-Driving Cars  With conventional cars, an owner that first purchased a new car will likely sell the car after a while. We all realize that a conventional car might end-up being passed from one buyer to another over its lifespan.  Will there be an equivalent market for used self-driving cars?  You might be inclined to immediately suggest that once a self-driving car has reached some point of no longer being safe enough, it needs to be retired. We don’t yet know, and no one has established what that safety juncture or threshold might be.  There could be a used self-driving car market that involved selling a used driverless car that was still within some bounds of being safe.  Suppose a driverless car owner that had used their self-driving car extensively in a downtown city setting opted to sell the autonomous car to someone that lived in a suburban community. The logic might be that the self-driving car no longer was sufficient for use in a stop-and-go traffic environment but might be viable in a less stressful suburban locale.  Overall, no one is especially thinking about used self-driving cars, which is admittedly a concern that is far away in the future and therefore not a topic looming over us today.  Retirement Of A Self-Driving Car  Other than becoming a used car, what else might happen to a self-driving car after it’s been in use for a while?  Some have wondered whether it might be feasible to convert a self-driving car into becoming a human-driven car, doing so to place the car into the used market for human-driven cars.  Well, it depends on how the self-driving car was originally made. If the self-driving car has all of the mechanical and electronic guts for human driving controls, you could presumably unplug the autonomy and revert the car into being a human-driven car.  I would assert that this is very unlikely, and you won’t see self-driving cars being transitioned into becoming human-driven cars.  All told, it would seem that once a self-driving car has reached its end of life, the vehicle would become scrapped.  If self-driving cars are being placed into the junk heap every four years, this raises the specter that we are going to have a lot of car junk piling up. For environmentalists, this is certainly disconcerting.  Generally, today’s cars are relatively highly recyclable and reusable. Estimates suggest that around 80% of a car can be recycled or reused.  For driverless cars, assuming they are built like today’s conventional cars, you would be able to potentially attain a similar recycled and reused parts percentage. The add-ons of the sensory devices and computer processors might be recyclable and reusable too, though this is not necessarily the case depending upon how the components were made.  For why remote piloting or operating of self-driving cars is generally eschewed, see my explanation here: https://aitrends.com/ai-insider/remote-piloting-is-a-self-driving-car-crutch/  To be wary of fake news about self-driving cars, see my tips here: https://aitrends.com/ai-insider/ai-fake-news-about-self-driving-cars/  The ethical implications of AI driving systems are significant, see my indication here: http://aitrends.com/selfdrivingcars/ethically-ambiguous-self-driving-cars/  Be aware of the pitfalls of normalization of deviance when it comes to self-driving cars, here’s my call to arms: https://aitrends.com/ai-insider/normalization-of-deviance-endangers-ai-self-driving-cars/  Conclusion  Some critics would be tempted to claim that the automakers would adore having self-driving cars that last only four years.  Presumably, it would mean that the automakers will be churning out new cars hand-over-fist, doing so to try and keep up with the demand for an ongoing supply of new driverless cars.  On the other hand, some pundits have predicted that we won’t need as many cars as we have today, since a smaller number of ridesharing driverless cars will fulfill our driving needs, abetting the need for everyone to have a car.  No one knows.  Another facet to consider involves the pace at which high-tech might advance and thus cause a heightened turnover in self-driving cars. Suppose the sensors and computer processors put into a driverless car are eclipsed in just a few years by faster, cheaper, and better sensors and computer processors.  If the sensors and processors of a self-driving car are built-in, meaning that you can’t just readily swap them out, it could be that another driving force for the quicker life cycle of a driverless car might be as a result of the desire to make use of the latest in high-tech.  The idea of retiring a driverless car in four years doesn’t seem quite as shocking after analyzing the basis for such a belief.  Whether society is better off or not as a result of self-driving cars, and also the matter of those self-driving cars only lasting four years, is a complex question. We’ll need to see how this all plays out.  Copyright 2020 Dr. Lance Eliot   This content is originally posted on AI Trends.  [Ed. Note: For reader’s interested in Dr. Eliot’s ongoing business analyses about the advent of self-driving cars, see his online Forbes column: https://forbes.com/sites/lanceeliot/]       
Read more
  • 0
  • 0
  • 3159

article-image-introducing-the-open-governance-network-model-from-linux-com
Matthew Emerick
15 Oct 2020
12 min read
Save for later

Introducing the Open Governance Network Model from Linux.com

Matthew Emerick
15 Oct 2020
12 min read
Background The Linux Foundation has long served as the home for many of the world’s most important open source software projects. We act as the vendor-neutral steward of the collaborative processes that developers engage in to create high quality and trustworthy code. We also work to build the developer and commercial communities around that code to sponsor each project’s members. We’ve learned that finding ways for all sorts of companies to benefit from using and contributing back to open source software development is key to the project’s sustainability. Over the last few years, we have also added a series of projects focused on lightweight open standards efforts — recognizing the critical complementary role that standards play in building the open technology landscape. Linux would not have been relevant if not for POSIX, nor would the Apache HTTPD server have mattered were it not for the HTTP specification. And just as with our open source software projects, commercial participants’ involvement has been critical to driving adoption and sustainability. On the horizon, we envision another category of collaboration, one which does not have a well-established term to define it, but which we are today calling “Open Governance Networks.” Before describing it, let’s talk about an example. Consider ICANN, the agency that arose after demands emerged from evolving the global domain name system (DNS) from its single-vendor control by Network Solutions. With ICANN, DNS became something more vendor-neutral, international, and accountable to the Internet community. It evolved to develop and manage the “root” of the domain name system, independent from any company or nation. ICANN’s control over the DNS comes primarily through its establishment of an operating agreement among domain name registrars that establishes rules for registrations, guarantees your domain names are portable, and a uniform dispute resolution protocol (the UDRP) for times when a domain name conflicts with an established trademark or causes other issues. ICANN is not a standards body; they happily use the standards for DNS developed at the IETF. They also do not create software other than software incidental to their mission, perhaps they also fund some DNS software development, but that’s not their core. ICANN is not where all DNS requests go to get resolved to IP addresses, nor even where everyone goes to register their domain name — that is all pushed to registrars and distributed name servers. In this way, ICANN is not fully decentralized but practices something you might call “minimum viable centralization.” Its management of the DNS has not been without critics, but by pushing as much of the hard work to the edge and focusing on being a neutral core, they’ve helped the DNS and the Internet achieve a degree of consistency, operational success, and trust that would have been hard to imagine building any other way. There are similar organizations that interface with open standards and software but perform governance functions. A prime example of this is the CA Browser Forum, who manages the root certificates for the SSL/TLS web security infrastructure. Do we need such organizations? Can’t we go completely decentralized? While some cryptocurrency networks claim not to need formal human governance, it’s clear that there are governance roles performed by individuals and organizations within those communities. Quite a bit of governance is possible to automate via smart contracts (and repairing damage from exploiting them), promoting the platform’s adoption to new users, onboarding new organizations, or even coordinating hard fork upgrades still require humans in the mix. And this is especially important in environments where competitors need to participate in the network to succeed, but do not trust one competitor to make the decisions. Network governance is not a solved problem Network governance is not just an issue for the technical layers. As one moves up the stack into more domain-specific applications, it turns out that there are network governance challenges up here as well, which look very familiar. Consider a typical distributed application pattern: supply chain traceability, where participants in the network can view, on a distributed database or ledger, the history of the movement of an object from source to destination, and update the network when they receive or send an object. You might be a raw materials supplier, or a manufacturer, or distributor, or retailer. In any case, you have a vested interest in not only being able to trust this distributed ledger to be an accurate and faithful representation of the truth. You also want the version you see to be the same ledger everyone else sees, be able to write to it fairly, and understand what happens if things go wrong. Achieving all of these desired characteristics requires network governance! You may be thinking that none of this is strictly needed if only everyone agreed to use one organization’s centralized database to serve as the system of record. Perhaps that is a company like eBay, or Amazon, Airbnb, or Uber. Or perhaps, a non-profit charity or government agency can run this database for us. There are some great examples of shared databases managed by non-profits, such as Wikipedia, run by the Wikimedia Foundation. This scenario might work for a distributed crowdsourced encyclopedia, but would it work for a supply chain? This participation model requires everyone engaging in the application ecosystem to trust that singular institution to perform a very critical role — and not be hacked, or corrupted, or otherwise use that position of power to unfair ends. There is also a trust the entity will not become insolvent or otherwise unable to meet the community’s needs. How many Wikipedia entries have been hijacked or subject to “edit wars” that go on forever? Could a company trust such an approach for its supply chain? Probably not. Over the last ten years, we’ve seen the development of new tools that allow us to build better-distributed data networks without that critical need for a centralized database or institution holding all the keys and trust. Most of these new tools use distributed ledger technology (“DLT”, or “blockchain”) to build a single source of truth across a network of cooperating peers, and embed programmatic functionality as “smart contracts” or “chaincode” across the network. The Linux Foundation has been very active in DLT, first with the launch of Hyperledger in December of 2015. The launch of the Trust Over IP Foundation earlier this year focused on the application of self-sovereign identity, and in many examples, usually using a DLT as the underlying utility network. As these efforts have focused on software, they left the development, deployment, and management of these DLT networks to others. Hundreds of such networks built on top of Hyperledger’s family of different protocol frameworks have launched, some of which (like the Food Trust Network) have grown to hundreds of participating organizations. Many of these networks were never intended to extend beyond an initial set of stakeholders, and they are seeing very successful outcomes. However, many of these networks need a critical mass of industry participants and have faced difficulty achieving their goal. A frequently cited reason is the lack of clear or vendor-neutral governance of the network. No business wants to place its data, or the data it depends upon, in the hands of a competitor; and many are wary even of non-competitors if it locks down competition or creates a dependency on a market participant. For example, what if the company doesn’t do well and decides to exit this business segment? And at the same time, for most applications, you need a large percentage of any given market to make it worthwhile, so addressing these kinds of business, risk, or political objections to the network structure is just as important as ensuring the software works as advertised. In many ways, this resembles the evolution of successful open source projects, where developers working at a particular company realize that just posting their source code to a public repository isn’t sufficient. Nor even is putting their development processes online and saying “patches welcome.” To take an open source project to the point where it becomes the reference solution for the problem being solved and can be trusted for mission-critical purposes, you need to show how its governance and sustainability are not dependent upon a single vendor, corporate largess, or charity. That usually means a project looks for a neutral home at a place like the Linux Foundation, to provide not just that neutrality, but also competent stewarding of the community and commercial ecosystem. Announcing LF Open Governance Networks To address this need, today, we are announcing that the Linux Foundation is adding “Open Governance Networks” to the types of projects we host. We have several such projects in development that will be announced before the end of the year. These projects will operate very similarly to the Linux Foundation’s open source software projects, but with some additional key functions. Their core activities will include: Hosting a technical steering committee to specify the software and standards used to build the network, to monitor the network’s health, and to coordinate upgrades, configurations, and critical bug fixes Hosting a policy and legal committee to specify a network operating agreement the organizations must agree to for connecting their nodes to the network Running a system for identity on the network, so participants to trust other participants who they say they are, monitor the network for health, and take corrective action if required. Building out a set of vendors who can be hired to deploy peers-as-a-service on behalf of members, in addition to allowing members’ technical staff to run their own if preferred. Convene a Governing Board composed of sponsoring members who oversee the budget and priorities. Advocate for the network’s adoption by the relevant industry, including engaging relevant regulators and secondary users who don’t run their own peers. Potentially manage an open “app store” approach to offering vetted re-usable deployable smart contracts of add-on apps for network users. These projects will be sustained through membership dues set by the Governing Board on each project, which will be kept to what’s needed for self-sufficiency. Some may also choose to establish transaction fees to compensate operators of peers if usage patterns suggest that would be beneficial. Projects will have complete autonomy regarding technical and software choices – there are no requirements to use other Linux Foundation technologies. To ensure that these efforts live up to the word “open” and the Linux Foundation’s pedigree, the vast majority of technical activity on these projects, and development of all required code and configurations to run the software that is core to the network will be done publicly. The source code and documentation will be published under suitable open source licenses, allowing for public engagement in the development process, leading to better long-term trust among participants, code quality, and successful outcomes. Hopefully, this will also result in less “bike-shedding” and thrash, better visibility into progress and activity, and an exit strategy should the cooperation efforts hit a snag. Depending on the industry that it services, the ledger itself might or might not be public. It may contain information only authorized for sharing between the parties involved on the network or account for GDPR or other regulatory compliance. However, we will certainly encourage long term approaches that do not treat the ledger data as sensitive. Also, an organization must be a member of the network to run peers on the network, required to see the ledger, and particularly write to it or participate in consensus. Across these Open Governance Network projects, there will be a shared operational, project management, marketing, and other logistical support provided by Linux Foundation personnel who will be well-versed in the platform issues and the unique legal and operational issues that arise, no matter which specific technology is chosen. These networks will create substantial commercial opportunity: For software companies building DLT-based applications, this will help you focus on the truly value-delivering apps on top of such a shared network, rather than the mechanics of forming these networks. For systems integrators, DLT integration with back-office databases and ERP is expected to grow to be billions of dollars in annual activity. For end-user organizations, the benefits of automating thankless, non-differentiating, perhaps even regulatorily-required functions could result in huge cost savings and resource optimization. For those organizations acting as governing bodies on such networks today, we can help you evolve those projects to reach an even wider audience while taking off your hands the low margin, often politically challenging, grunt work of managing such networks. And for those developers concerned before about whether such “private” permissioned networks would lead to dead cul-de-sacs of software and wasted effort or lost opportunity, having the Linux Foundation’s bedrock of open source principles and collaboration techniques behind the development of these networks should help ensure success. We also recognize that not all networks should be under this model. We expect a diversity of approaches that will be long term sustainable, and encourage these networks to find a model that works for them. Let’s talk to see if it would be appropriate. LF Governance Networks will enable our communities to establish their own Open Governance Network and have an entity to process agreements and collect transaction fees. This new entity is a Delaware nonprofit, a nonstock corporation that will maximize utility and not profit. Through agreements with the Linux Foundation, LF Governance Networks will be available to Open Governance Networks hosted at the Linux Foundation. If you’re interested in learning more about hosting an Open Governance Network at the Linux Foundation, please contact us at governancenetworks@linuxfoundation.org Thanks! Brian The post Introducing the Open Governance Network Model appeared first on The Linux Foundation. The post Introducing the Open Governance Network Model appeared first on Linux.com.
Read more
  • 0
  • 0
  • 999

article-image-how-to-use-java-generics-to-avoid-classcastexceptions-from-infoworld-java
Matthew Emerick
15 Oct 2020
1 min read
Save for later

How to use Java generics to avoid ClassCastExceptions from InfoWorld Java

Matthew Emerick
15 Oct 2020
1 min read
Java 5 brought generics to the Java language. In this article, I introduce you to generics and discuss generic types, generic methods, generics and type inference, generics controversy, and generics and heap pollution. download Get the code Download the source code for examples in this Java 101 tutorial. Created by Jeff Friesen for JavaWorld. What are generics? Generics are a collection of related language features that allow types or methods to operate on objects of various types while providing compile-time type safety. Generics features address the problem of java.lang.ClassCastExceptions being thrown at runtime, which are the result of code that is not type safe (i.e., casting objects from their current types to incompatible types). To read this article in full, please click here
Read more
  • 0
  • 0
  • 5241
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at $19.99/month. Cancel anytime
article-image-mikroorm-4-1-lets-talk-about-performance-from-dailyjs-medium
Matthew Emerick
15 Oct 2020
3 min read
Save for later

MikroORM 4.1: Let’s talk about performance from DailyJS - Medium

Matthew Emerick
15 Oct 2020
3 min read
I just shipped version 4.1 of MikroORM, the TypeScript ORM for Node.js, and I feel like this particular release deserves a bit more attention than a regular feature release. In case you don’t know… If you never heard of MikroORM, it’s a TypeScript data-mapper ORM with Unit of Work and Identity Map. It supports MongoDB, MySQL, PostgreSQL and SQLite drivers currently. Key features of the ORM are: Implicit transactions ChangeSet based persistence Identity map You can read the full introductory article here or browse through the docs. So what changed? This release had only one clear goal in mind — the performance. It all started with an issue pointing out that flushing 10k entities in a single unit of work is very slow. While this kind of use case was never a target for me, I started to see all the possibilities the Unit of Work pattern offers. Batch inserts, updates and deletes The biggest performance killer was the amount of queries — even if the query is as simple and optimised as possible, firing 10k of those will be always quite slow. For inserts and deletes, it was quite trivial to group all the queries. A bit more challenging were the updates — to batch those, MikroORM now uses case statements. As a result, when you now flush changes made to one entity type, only one query per given operation (create/update/delete) will be executed. This brings significant difference, as we are now executing fixed number of queries (in fact the changes are batched in chunks of 300 items). https://medium.com/media/3df9aaa8c2f0cf018855bf66ecf3d065/href JIT compilation Second important change in 4.1 is JIT compilation. Under the hood, MikroORM now first generates simple functions for comparing and hydrating entities, that are tailored to their metadata definition. The main difference is that those generated functions are accessing the object properties directly (e.g. o.name), instead of dynamically (e.g. o[prop.name]), as all the information from metadata are inlined there. This allows V8 to better understand the code so it is able to run it faster. Results Here are the results for a simple 10k entities benchmark: In average, inserting 10k entities takes around 70ms with sqlite, updates are a tiny bit slower. You can see results for other drivers here: https://github.com/mikro-orm/benchmark. Acknowledgement Kudos to Marc J. Schmidt, the author of the initial issue, as without his help this would probably never happen, or at least not in near future. Thanks a lot! Like MikroORM? ⭐️ Star it on GitHub and share this article with your friends. If you want to support the project financially, you can do so via GitHub Sponsors. MikroORM 4.1: Let’s talk about performance was originally published in DailyJS on Medium, where people are continuing the conversation by highlighting and responding to this story.
Read more
  • 0
  • 0
  • 5417

article-image-image-processing-techniques-that-you-can-use-in-machine-learning-projects-planet-scipy
Matthew Emerick
15 Oct 2020
1 min read
Save for later

Image Processing Techniques That You Can Use in Machine Learning Projects - Planet SciPy

Matthew Emerick
15 Oct 2020
1 min read
Image processing is a method to perform operations on an image to extract information from it or enhance it. Digital image processing... The post Image Processing Techniques That You Can Use in Machine Learning Projects appeared first on neptune.ai.
Read more
  • 0
  • 0
  • 2053

Matthew Emerick
15 Oct 2020
1 min read
Save for later

Has the Time Come for IT Predictive Analytics? from DevOps.com

Matthew Emerick
15 Oct 2020
1 min read
Predictive analytics technologies have become critical to compete in manufacturing (predicting machine failure), banking (predicting fraud), e-commerce (predicting buying behavior) as well as to address horizontal use cases such as cybersecurity breach prevention and sales forecasting. Using data to predict and prevent IT outages and issues is also a growing best practice—especially as advances in […] The post Has the Time Come for IT Predictive Analytics? appeared first on DevOps.com.
Read more
  • 0
  • 0
  • 2480

article-image-apache-spark-on-kubernetes-how-apache-yunikorn-incubating-helps-from-cloudera-blog
Matthew Emerick
14 Oct 2020
10 min read
Save for later

Apache Spark on Kubernetes: How Apache YuniKorn (Incubating) helps from Cloudera Blog

Matthew Emerick
14 Oct 2020
10 min read
Background Why choose K8s for Apache Spark Apache Spark unifies batch processing, real-time processing, stream analytics, machine learning, and interactive query in one-platform. While Apache Spark provides a lot of capabilities to support diversified use cases, it comes with additional complexity and high maintenance costs for cluster administrators. Let’s look at some of the high-level requirements for the underlying resource orchestrator to empower Spark as a one-platform: Containerized Spark compute to provide shared resources across different ML and ETL jobs Support for multiple Spark versions, Python versions, and version-controlled containers on the shared K8s clusters for both faster iteration and stable production A single, unified infrastructure for both majority of batch workloads and microservices Fine-grained access controls on shared clusters Kubernetes as a de-facto standard for service deployment offers finer control on all of the above aspects compared to other resource orchestrators. Kubernetes offers a simplified way to manage infrastructure and applications with a practical approach to isolate workloads, limit the use of resources, deploy on-demand resources, and auto-scaling capabilities as needed. Scheduling challenges to run Apache Spark on K8s Kubernetes default scheduler has gaps in terms of deploying batch workloads efficiently in the same cluster where long-running services are also to be scheduled. Batch workloads need to be scheduled mostly together and much more frequently due to the nature of compute parallelism required. Let’s look at some of those gaps in detail. Lack of first-class application concept  Batch jobs often need to be scheduled in a sequential manner based on types of container deployment. For instance, Spark driver pods need to be scheduled earlier than worker pods. A clear first-class application concept could help with ordering or queuing each container deployment. Also, such a concept helps admin to visualize the jobs which are scheduled for debugging purposes. Lack of efficient capacity/quota management capability  Kubernetes namespace resource quota can be used to manage resources while running a Spark workload in multi-tenant use cases. However, there are few challenges in achieving this, Apache Spark jobs are dynamic in nature with regards to their resource usage. Namespace quotas are fixed and checked during the admission phase. The pod request is rejected if it does not fit into the namespace quota. This requires the Apache Spark job to implement a retry mechanism for pod requests instead of queueing the request for execution inside Kubernetes itself.  The namespace resource quota is flat, it doesn’t support hierarchy resource quota management. Also many a time, user’s could starve to run the batch workloads as Kubernetes namespace quotas often do not match the organizational hierarchy based capacity distribution plan. An elastic and hierarchical priority management for jobs in K8s is missing today. Lack of resource fairness between tenants In a production environment, it is often found that Kubernetes default scheduler could not efficiently manage diversified workloads and provide resource fairness for their workloads. Some of the key reasons are: Batch workload management in a production environment will often be running with a large number of users. In a dense production environment where different types of workloads are running, it is highly possible that Spark driver pods could occupy all resources in a namespace. Such scenarios pose a big challenge in effective resource sharing.  Abusive or corrupted jobs could steal resources easily and impact production workloads. Strict SLA requirements with scheduling latency  Most of the busy production clusters dedicated for batch workloads often run thousands of jobs with hundreds of thousands of tasks every day. These workloads require larger amounts of parallel container deployments and often the lifetime of such containers is short (from seconds to hours). This usually produces a demand for thousands of pod or container deployment waiting to be scheduled, using Kubernetes default scheduler can introduce additional delays which could lead to missing of SLAs. How Apache YuniKorn (Incubating) could help Overview of Apache YuniKorn (Incubating)  YuniKorn is an enhanced Kubernetes scheduler for both services and batch workloads. YuniKorn can replace Kubernetes default scheduler, or also work with K8s default scheduler based on the deployment use cases. YuniKorn brings a unified, cross-platform scheduling experience for mixed workloads consisting of stateless batch workloads and stateful services. YuniKorn v.s. Kubernetes default scheduler: A comparison   Feature Default Scheduler YUNIKORN Note Application concept x √ Applications are a 1st class citizen in YuniKorn. YuniKorn schedules apps with respect to, e,g their submission order, priority, resource usage, etc. Job ordering x √ YuniKorn supports FIFO/FAIR/Priority (WIP) job ordering policies Fine-grained resource capacity management x √ Manage cluster resources with hierarchy queues. Queues provide the guaranteed resources (min) and the resource quota limit (max). Resource fairness x √ Resource fairness across application and queues to get ideal allocation for all applications running Natively support Big Data workloads x √ Default scheduler focuses for long-running services. YuniKorn is designed for Big Data app workloads, and it natively supports to run Spark/Flink/Tensorflow, etc efficiently in K8s. Scale  & Performance x √ YuniKorn is optimized for performance, it is suitable for high throughput and large scale environments. How YuniKorn helps to run Spark on K8s YuniKorn has a rich set of features that help to run Apache Spark much efficiently on Kubernetes. Detailed steps can be found here to run Spark on K8s with YuniKorn. Please read more details about how YuniKorn empowers running Spark on K8s in Cloud-Native Spark Scheduling with YuniKorn Scheduler in Spark & AI summit 2020. Let’s take a look at some of the use cases and how YuniKorn helps to achieve better resource scheduling for Spark in these scenarios. Multiple users (noisy) running different spark workloads together As more users start to run jobs together, it becomes very difficult to isolate and provide required resources for the jobs with resource fairness, priority etc. YuniKorn scheduler provides an optimal solution to manage resource quotas by using resource queues. In the above example of a queue structure in YuniKorn, namespaces defined in Kubernetes are mapped to queues under the Namespaces parent queue using a placement policy. The Test and Development queue have fixed resource limits. All other queues are only limited by the size of the cluster. Resources are distributed using a Fair policy between the queues, and jobs are scheduled FIFO in the production queue. Some of the high-level advantages are, One YuniKorn queue can map to one namespace automatically in Kubernetes Queue Capacity is elastic in nature which could provide resource range from a configured min to max value Honor resource fairness which could avoid possible resource starvation YuniKorn provides a seamless way to manage resource quota for a Kubernetes cluster, it can work as a replacement of the namespace resource quota. YuniKorn resource quota management allows leveraging queuing of pod requests and sharing of limited resources between jobs based on pluggable scheduling policies. This all can be achieved without any further requirements, like retrying pod submits, on Apache Spark. Setting up the cluster to organization hierarchy based resource allocation model In a large production environment, multiple users will be running various types of workloads together. Often these users are bound to consume resources based on the organization team hierarchy budget constraints. Such a production setup helps for efficient cluster resource usage within resource quota boundaries. YuniKorn provides an ability to manage resources in a cluster with a hierarchy of queues. A fine-grained resource capacity management for a multi-tenant environment will be possible by using resource queues with clear hierarchy (like organization hierarchy). YuniKorn queues can be statically configured or dynamically managed and with the dynamic queue management feature, users can set up placement rules to delegate queue management. Better Spark job SLA in a multi-tenant cluster Normal ETL workloads running in a multi-tenant cluster require easier means of defining fine-grained policies to run jobs in the desired organizational queue hierarchy. Many times, such policies help to define stricter SLA’s for job execution. YuniKorn empowers administrators with options to enable the Job ordering in queues based on simpler policies such as FIFO, FAIR, etc. The StateAware app sorting policy orders jobs in a queue in FIFO order and schedules them one by one on conditions. This avoids the common race condition while submitting lots of batch jobs, e.g Spark, to a single namespace (or cluster). By enforcing the specific ordering of jobs, it also improves the scheduling of jobs to be more predictable. Enable various K8s feature sets for Apache Spark Job scheduling YuniKorn is fully compatible with K8s major released versions. Users can swap the scheduler transparently on an existing K8s cluster. YuniKorn fully supports all the native K8s semantics that can be used during scheduling, such as label selector, pod affinity/anti-affinity, taints/toleration, PV/PVCs, etc. YuniKorn is also compatible with the management commands and utilities, such as cordon nodes, retrieving events via kubectl, etc. Apache YuniKorn (Incubating) in CDP Cloudera’s CDP platform offers Cloudera Data Engineering experience which is powered by Apache YuniKorn (Incubating). Some of the high-level use cases solved by YuniKorn at Cloudera are, Provide resource quota management for CDE virtual clusters Provide advanced job scheduling capabilities for Spark Responsible for both micro-service and batch jobs scheduling Running on Cloud with auto-scaling enabled Future roadmaps to better support Spark workloads YuniKorn community is actively looking into some of the core feature enhancements to support Spark workloads execution. Some of the high-level features are For Spark workloads, it is essential that a minimum number of driver & worker pods be allocated for better efficient execution. Gang scheduling helps to ensure a required number of pods be allocated to start the Spark job execution. Such a feature will be very helpful in a noisy multi-tenant cluster deployment. For more details, YUNIKORN-2 Jira is tracking the feature progress. Job/Task priority support Job level priority ordering helps admin users to prioritize and direct YuniKorn to provision required resources for high SLA based job execution. This also gives more flexibility for effective usage of cluster resources. For more details, YUNIKORN-1 Jira is tracking the feature progress. Distributed Tracing YUNIKORN-387 leverages Open Tracing to improve the overall observability of the scheduler. With this feature, the critical traces through the core scheduling cycle can be collected and persisted for troubleshooting, system profiling, and monitoring. Summary YuniKorn helps to achieve fine-grained resource sharing for various Spark workloads efficiently on a large scale, multi-tenant environments on one hand and dynamically brought up cloud-native environments on the other. YuniKorn, thus empowers Apache Spark to become an enterprise-grade essential platform for users, offering a robust platform for a variety of applications ranging from large scale data transformation to analytics to machine learning. Acknowledgments Thanks to Shaun Ahmadian and Dale Richardson for reviewing and sharing comments. A huge thanks to YuniKorn open source community members who helped to get these features to the latest Apache release. The post Apache Spark on Kubernetes: How Apache YuniKorn (Incubating) helps appeared first on Cloudera Blog.
Read more
  • 0
  • 0
  • 1864
article-image-using-cloudera-machine-learning-to-build-a-predictive-maintenance-model-for-jet-engines-from-cloudera-blog
Matthew Emerick
14 Oct 2020
6 min read
Save for later

Using Cloudera Machine Learning to Build a Predictive Maintenance Model for Jet Engines from Cloudera Blog

Matthew Emerick
14 Oct 2020
6 min read
Introduction Running a large commercial airline requires the complex management of critical components, including fuel futures contracts, aircraft maintenance and customer expectations. Airlines, in just the U.S. alone, average about 45,000 daily flights, transporting over 10 million passengers a year (source: FAA). Airlines typically operate on very thin margins, and any schedule delay immediately angers or frustrates customers. Flying is not inherently dangerous, but the consequence of a failure is catastrophic. Airlines have such a sophisticated business model that encompasses a culture of streamlined supply chains, predictive maintenance, and unwavering customer satisfaction. To maximize safety for all passengers and crew members, while also delivering profits, airlines have heavily invested in predictive analytics to gain insight on the most cost-effective way to maintain real-time engine performance. Additionally, airlines ensure availability and reliability of their fleet by leveraging maintenance, overhaul and repair (MRO) organizations, such as Lufthansa Technik.  Lufthansa Technik worked with Cloudera to build a predictive maintenance platform to service its fleet of 5000 aircraft throughout its global network of 800 MRO facilities. Lufthansa Technik extended a standard practice of placing sensors on aircraft engines and enabling predictive maintenance to automate fulfilment solutions. By combining profound airline operation expertise, data science, and engine analytics to a predictive maintenance schedule, Lufthansa Technik can now ensure critical parts are on the ground (OTG) when needed, instead of the entire aircraft being OTG and not producing revenue. The objective of this blog is to show how to use Cloudera Machine Learning (CML), running Cloudera Data Platform (CDP), to build a predictive maintenance model based on advanced machine learning concepts. The Process Many companies build machine learning models using libraries, whether they are building perception layers for autonomous vehicles, allowing autonomous vehicle operation, or modeling a complex jet engine. Kaggle, a site that provides test training data sets for building machine learning models, provides simulation data sets from NASA that measures engine component degradation for turbofan jet engines. The models in this blog are built on CML and are based on inputting various engine parameters showing typical sensor values of engine temperature, fuel consumption, vibration, or fuel to oxygen mixture (see Fig 1). One item to note in this blog is that the term “failure” is not to imply catastrophic failure, but rather, that one of its components (pumps, values, etc) is not operating to specification. Airlines design their aircraft to operate at 99.999% reliability. Fig 1: Turbofan jet engine Step 1: Using the training data to create a model/classifier First, four test and training data sets for varying conditions and failure modes were organized in preparation for CML (see box 1 in Fig 2). Each set of training data shows the engine parameters per flight while each engine is “flown” until an engine component signals failure. This is done at both sea level and all flight conditions. This data will be used to train the model that can predict how many flights a given engine has until failure. For each training set, there is a corresponding test data set that provides data on 100 jet engines at various stages of life with actual values on which to test the predictive model for accuracy.  Fig 2: Diagram showing how CML is used to build ML training models Step 2: Iterate on the model to validate and improve effectiveness CML was used to create a model that estimated the amount of remaining useful life (RUL) for a given engine using the provided test and training data sets. A threshold of one week–the time allowance to place parts on the ground–was planned for a scenario that alerts an airline before a potential engine component failure. Assuming four flights daily, this means the airline would like to know with confidence if an engine is going to fail within 40 flights. The model was tested for each engine, and the results were classified as true or false for potential failure within 40 flights (see Table 1). Table 1: Data in table based on one week of data of 40 flights. Step 3: Apply an added cost value to the results With no preventative maintenance, an engine that runs out of life or fails can compromise safety and cost millions more dollars to replace an engine. If an engine is maintained or overhauled before it runs out of life, the cost of overhaul is significantly less. However, if the engine is overhauled too early, there is potential engine life that could have still been utilized. The estimated cost in this model for each of these overhaul outcomes can be seen below (see Fig 3). Fig 3: Cost-benefit confusion matrix Conclusion Using Cloudera Machine Learning to analyze NASA jet engine simulation data provided by Kaggle, our predictive maintenance model predicted when an engine was likely to fail or when it required an overhaul with very high accuracy. Combining the cost-benefit analysis with this predictive model against the test data sets suggested significant savings across all applied scenarios. Airline decisions are always made with a consideration to safety first and then consideration to profit second. Predictive maintenance is preferred because it is always the safest choice, and it delivers drastically lower maintenance costs over reactive (engine replacement after failure) or proactive (replacing components before engine replacement) approaches. Next Steps To see all this in action, please click on links below to a few different sources showcasing the process that was created. Video – If you’d like to see and hear how this was built, see video at the link. Tutorials – If you’d like to do this at your own pace, see a detailed walkthrough with screenshots and line by line instructions of how to set this up and execute. Meetup – If you want to talk directly with experts from Cloudera, please join a virtual meetup to see a live stream presentation. There will be time for direct Q&A at the end. CDP Users Page – To learn about other CDP resources built for users, including additional video, tutorials, blogs and events, click on the link. The post Using Cloudera Machine Learning to Build a Predictive Maintenance Model for Jet Engines appeared first on Cloudera Blog.
Read more
  • 0
  • 0
  • 1789

article-image-election-2020-how-data-can-show-whats-driving-the-trump-vs-biden-polls-from-whats-new
Matthew Emerick
14 Oct 2020
8 min read
Save for later

Election 2020: How data can show what’s driving the Trump vs. Biden polls from What's New

Matthew Emerick
14 Oct 2020
8 min read
Steve Schwartz Director, Public Affairs at Tableau Tanna Solberg October 14, 2020 - 5:19pm October 14, 2020 It’s October, which means there is officially less than one month until the 2020 Presidential election on November 3. Opinion polls on the race between current President Donald Trump and the challenger, former Vice President Joe Biden, are everywhere.  Although many people see public opinion polls as a way to anticipate the outcome of the election, they are most valuable when considered as a snapshot of people's beliefs at a given moment in time. Through our partnership with SurveyMonkey and Axios to collect and share data on the 2020 Presidential race, we’ve created a dashboard where you can track how survey respondents are feeling about the candidates. But looking at candidate preference data alone doesn’t answer the critical question of this years’ election: What is driving voter preference? This year, that’s an especially tricky question. There are the major issues confronting the country this year—from challenges like the COVID-19 pandemic, to the disease’s impact on the national and global economies, to the nationwide protests for racial justice and equity. And there’s also the news cycle which seemingly tosses another knuckleball at voters before they’ve had a chance to process the last one. By partnering with SurveyMonkey, we’ve been able to tap into their vast market research technologies to reach the public and visualize their answers to these critical questions. Through our Election 2020 platform, you can dig into this data and expand your understanding of not only what the topline polls are saying, but what is top-of-mind for the voters making the decision this year. We’ll walk you through some of the key data you can find on our Election 2020 pages, and why it’s so critical to understanding this year’s political landscape. Preferences by demographics Understanding the way different demographic groups vote is critical. It’s very common for pollsters to break down results by categories like age bracket, race, and gender: Disaggregating data offers valuable insights into trends among voter groups that can inform understanding of potential election results. But the way the data is often presented—either in static crosstabs, or in percentage points scattered throughout an analysis—doesn’t really give people the insight into voters’ intersectionalities, and how they play out in the data. SurveyMonkey wanted to give people a way to explore demographic data in a more granular and comprehensive way. They’ve broken down data on candidate preference by five different demographic categories—age, race, gender, education level, and party ID—and in a Tableau dashboard, anyone can choose which categories to combine to see more nuanced voter preferences. For instance, if one were to just look at gender, the breakdown would be pretty clear: 52% of men support Trump, and 56% of women support Biden. But in this dashboard, you can also choose to layer in race. Suddenly, the picture becomes much more complex: 87% of Black women support Biden, and 76% of Black men support Biden. On the flip side, just 38% of white men support Biden, and 44% of white women support Trump. If you add in another dimension, like education, the numbers become even clearer: By far, the group that most strongly supports Trump (at 70%) is white men with a high school degree or less, and the group with the strongest across-the-board support for Biden is Black women with a postgraduate degree (91%). "From the perspective of someone who's immersed in crosstabs and bar charts every day, this visualization is the clearest example yet of the value of pairing data collected through SurveyMonkey's mighty scale with visual storytelling tools from Tableau. The fact that it's highly interactive and responsive really brings the data to life in a way that isn't possible using standard tools,” Wronski says. The COVID-19 pandemic Let’s start with the big one. COVID-19 has posed one of the most significant challenges to the United States and its citizens in recent memory. Over 200,000 people have died, and the economy has recorded its steepest-ever drop on record, with the GDP declining more than 9%. As we near the Election, the virus is not showing signs of abating (for the latest data on COVID-19, you can visit Tableau’s COVID-19 Data Hub). Our partners at SurveyMonkey have been tracking public sentiment around the pandemic since February, as it’s impacting the lives of nearly everyone in the United States this year. "The coronavirus pandemic has infiltrated every aspect of life for the past eight months, and it will continue to do so for the foreseeable future. We wanted to make sure to start measuring concerns early on, and we're committed to tracking public sentiment on this topic for as long as necessary,” says Laura Wronski, research science manager at SurveyMonkey. Through our Election 2020 portal, you can analyze data on how the public is feeling about the pandemic in the leadup to the election. SurveyMonkey has asked respondents about their personal concerns around the virus—if they are worried about contracting it themselves, or someone in their family being affected, and if they are worried about the pandemic’s impact on the economy. Because SurveyMonkey’s Tableau dashboards make it easy to filter these responses by a number of demographic factors—from age to political affiliation—you can begin to see patterns in the data, and understand how concerns around COVID-19 could be a key factor in shaping the outcome of the election. Government leadership Elections are nearly always a referendum on leadership, and this year is no different. However, the pandemic is adding a new layer to how voters assess their elected leaders across the country. "As the election approaches, politicians who are on the ballot at every level will be judged by how well they responded to the coronavirus this year, both in terms of its effect on the economy through lost jobs and shuttered businesses and in terms of the public health infrastructure's response,” Wronski says. Digging into the data, you can see virtually no difference along any demographic breakdown between people’s assessment of Trump as a leader overall, and people’s opinion of how he is handling the federal response to COVID-19. That can tell you several things: That voters’ opinions are, at this point, fairly solidified, and also that COVID-19 is a significant driver of that opinion. Digging into the data on how respondents feel about their state government’s response to COVID-19 shows some interesting trends. The clearest split, in many states, seems to be along party lines. In Pennsylvania, for instance, 82% of Democrats approve of the state response, while 71% of Republicans disapprove. In South Carolina, 73% of Republicans approve of the response, and 74% of Republicans disapprove. It gets much more interesting along other demographic lines, though. Here’s the opinion split along gender lines in Pennsylvania: 60% of women approve, and 49% of men disapprove. And in South Carolina, 54% of women disapprove, and 54% of men approve. "Like so much else these days, Republicans and Democrats are split in their views of how worrisome the coronavirus is and how well we've responded to it. Those partisan effects far outweigh any differences by age, gender, race, or other demographic characteristics,” Wronski says. Voting COVID-19 has complicated nearly every aspect of the 2020 Election, including voting. Multiple news outlets are reporting a sharp uptick in requests to vote by mail this year, due to concerns about gathering in public amid a pandemic. But through their data on how likely people are to vote by mail, SurveyMonkey is able to show a clear split along party lines. Overall, 70% of Democrat respondents say they’re likely to vote by mail, and 72% of Republican respondents say the opposite. Axios, our media partner in our Election 2020 initiative, has analyzed what this means in the context of the potential outcome, and what the implications could be if mail-in ballots are disqualified due to complications with the system. "More people will vote by mail in this election than in any previous election, and that will reshape the logistics of the electoral tallying process and the entire narrative that we see on the news on Election Day. It's important for us to understand those dynamics early on so that we can help explain those changes to the public,” Wronski says. Exploring with data Now that you have a sense of the information SurveyMonkey is polling for and why—and how to discover it in Tableau—we hope you take some time to dig into the data and gather your own insights. As the election nears, SurveyMonkey, Tableau, and Axios will continue to deliver more data and analysis around the political landscape, so make sure you keep checking back to the Election 2020 page for the latest.
Read more
  • 0
  • 0
  • 930

article-image-tom-swartz-tuning-your-postgres-database-for-high-write-loads-from-planet-postgresql
Matthew Emerick
14 Oct 2020
1 min read
Save for later

Tom Swartz: Tuning Your Postgres Database for High Write Loads from Planet PostgreSQL

Matthew Emerick
14 Oct 2020
1 min read
As a database grows and scales up from a proof of concept to a full-fledged production instance, there are always a variety of growing pains that database administrators and systems administrators will run into. Very often, the engineers on the Crunchy Data support team help support enterprise projects which start out as small, proof of concept systems, and are then promoted to large scale production uses.  As these systems receive increased traffic load beyond their original proof-of-concept sizes, one issue may be observed in the Postgres logs as the following: LOG: checkpoints are occurring too frequently (9 seconds apart) HINT: Consider increasing the configuration parameter "max_wal_size". LOG: checkpoints are occurring too frequently (2 seconds apart) HINT: Consider increasing the configuration parameter "max_wal_size".
Read more
  • 0
  • 0
  • 1297
article-image-why-congress-should-invest-in-open-source-software-brookings-from-linux-com
Matthew Emerick
14 Oct 2020
2 min read
Save for later

Why Congress should invest in open-source software (Brookings) from Linux.com

Matthew Emerick
14 Oct 2020
2 min read
Frank Nagle at Brookings writes: As the pandemic has highlighted, our economy is increasingly reliant on digital infrastructure. As more and more in-person interactions have moved online, products like Zoom have become critical infrastructure supporting business meetings, classroom education, and even congressional hearings. Such communication technologies build on FOSS and rely on the FOSS that is deeply ingrained in the core of the internet. Even grocery shopping, one of the strongholds of brick and mortar retail, has seen an increased reliance on digital technology that allows higher-risk shoppers to pay someone to shop for them via apps like InstaCart (which itself relies on, and contributes to, FOSS). As the pandemic has highlighted, our economy is increasingly reliant on digital infrastructure. As more and more in-person interactions have moved online, products like Zoom have become critical infrastructure supporting business meetings, classroom education, and even congressional hearings. Such communication technologies build on FOSS and rely on the FOSS that is deeply ingrained in the core of the internet. Even grocery shopping, one of the strongholds of brick and mortar retail, has seen an increased reliance on digital technology that allows higher-risk shoppers to pay someone to shop for them via apps like InstaCart (which itself relies on, and contributes to, FOSS). Read more at Brookings The post Why Congress should invest in open-source software (Brookings) appeared first on Linux.com.
Read more
  • 0
  • 0
  • 948

Matthew Emerick
14 Oct 2020
1 min read
Save for later

Bruce Momjian: Thirty Years of Continuous PostgreSQL Development from Planet PostgreSQL

Matthew Emerick
14 Oct 2020
1 min read
I did an interview with EDB recently, and a blog post based on that interview was published yesterday. It covers the Postgres 13 feature set and the effects of open source on the software development process.
Read more
  • 0
  • 0
  • 1680