Artificial intelligence for augmentation and productivity

Posted on August 23, 2023 by - News

The MIT Stephen A. Schwarzman College of Computing has awarded seed grants to seven projects that are exploring how artificial intelligence and human-computer interaction can be leveraged to enhance modern work spaces to achieve better management and higher productivity.

Funded by Andrew W. Houston ’05 and Dropbox Inc., the projects are intended to be interdisciplinary and bring together researchers from computing, social sciences, and management.

The seed grants can enable the project teams to conduct research that leads to bigger endeavors in this rapidly evolving area, as well as build community around questions related to AI-augmented management.

The seven selected projects and research leads include:

“LLMex: Implementing Vannevar Bush’s Vision of the Memex Using Large Language Models,” led by Pattie Maes of the Media Lab and David Karger of the Department of Electrical Engineering and Computer Science (EECS) and the Computer Science and Artificial Intelligence Laboratory (CSAIL). Inspired by Vannevar Bush’s Memex, this project proposes to design, implement, and test the concept of memory prosthetics using large language models (LLMs). The AI-based system will intelligently help an individual keep track of vast amounts of information, accelerate productivity, and reduce errors by automatically recording their work actions and meetings, supporting retrieval based on metadata and vague descriptions, and suggesting relevant, personalized information proactively based on the user’s current focus and context.

“Using AI Agents to Simulate Social Scenarios,” led by John Horton of the MIT Sloan School of Management and Jacob Andreas of EECS and CSAIL. This project imagines the ability to easily simulate policies, organizational arrangements, and communication tools with AI agents before implementation. Tapping into the capabilities of modern LLMs to serve as a computational model of humans makes this vision of social simulation more realistic, and potentially more predictive.

“Human Expertise in the Age of AI: Can We Have Our Cake and Eat it Too?” led by Manish Raghavan of MIT Sloan and EECS, and Devavrat Shah of EECS and the Laboratory for Information and Decision Systems. Progress in machine learning, AI, and in algorithmic decision aids has raised the prospect that algorithms may complement human decision-making in a wide variety of settings. Rather than replacing human professionals, this project sees a future where AI and algorithmic decision aids play a role that is complementary to human expertise.

“Implementing Generative AI in U.S. Hospitals,” led by Julie Shah of the Department of Aeronautics and Astronautics and CSAIL, Retsef Levi of MIT Sloan and the Operations Research Center, Kate Kellog of MIT Sloan, and Ben Armstrong of the Industrial Performance Center. In recent years, studies have linked a rise in burnout from doctors and nurses in the United States with increased administrative burdens associated with electronic health records and other technologies. This project aims to develop a holistic framework to study how generative AI technologies can both increase productivity for organizations and improve job quality for workers in health care settings.

“Generative AI Augmented Software Tools to Democratize Programming,” led by Harold Abelson of EECS and CSAIL, Cynthia Breazeal of the Media Lab, and Eric Klopfer of the Comparative Media Studies/Writing. Progress in generative AI over the past year is fomenting an upheaval in assumptions about future careers in software and deprecating the role of coding. This project will stimulate a similar transformation in computing education for those who have no prior technical training by creating a software tool that could eliminate much of the need for learners to deal with code when creating applications.

“Acquiring Expertise and Societal Productivity in a World of Artificial Intelligence,” led by David Atkin and Martin Beraja of the Department of Economics, and Danielle Li of MIT Sloan. Generative AI is thought to augment the capabilities of workers performing cognitive tasks. This project seeks to better understand how the arrival of AI technologies may impact skill acquisition and productivity, and to explore complementary policy interventions that will allow society to maximize the gains from such technologies.

“AI Augmented Onboarding and Support,” led by Tim Kraska of EECS and CSAIL, and Christoph Paus of the Department of Physics and the Laboratory for Nuclear Science. While LLMs have made enormous leaps forward in recent years and are poised to fundamentally change the way students and professionals learn about new tools and systems, there is often a steep learning curve which people have to climb in order to make full use of the resource. To help mitigate the issue, this project proposes the development of new LLM-powered onboarding and support systems that will positively impact the way support teams operate and improve the user experience.

M’Care and MIT students join forces to improve child health in Nigeria

Posted on August 23, 2023 by - News

Through a collaboration between M’Care, a 2021 Health Security and Pandemics Solver team, and students from MIT, the landscape of child health care in Nigeria could undergo a transformative change, wherein the power of data is harnessed to improve child health outcomes in economically disadvantaged communities. 

M’Care is a mobile application of Promane and Promade Limited, developed by Opeoluwa Ashimi, which gives community health workers in Nigeria real-time diagnostic and treatment support. The application also creates a dashboard that is available to government health officials to help identify disease trends and deploy timely interventions. As part of its work, M’Care is working to mitigate malnutrition by providing micronutrient powder, vitamin A, and zinc to children below the age of 5. To help deepen its impact, Ashimi decided to work with students in the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) course 6.S897 (Machine Learning for Healthcare) — instructed by professors Peter Szolovits and Manolis Kellis — to leverage data in order to improve nutrient delivery to children across Nigeria. The collaboration also enabled students to see real-world applications for data analysis in the health care space.

A meeting of minds: M’Care, MIT, and national health authorities

“Our primary goal for collaborating with the ML for Health team was to spot the missing link in the continuum of care. With over 1 million cumulative consultations that qualify for a continuum of care evaluation, it was important to spot why patients could be lost to followup, prevent this, and ensure completion of care to successfully address the health needs of our patients,” says Ashimi, founder and CEO of M’Care.

In May 2023, Ashimi attended a meeting that brought together key national stakeholders, including the representatives of the National Ministry of Health in Nigeria. This gathering served as a platform to discuss the profound impact of M’Care’s and ML for Health team’s collaboration — bolstered by data analysis provided on dosage regimens and a child’s age to enhance continuum of care with its attendant impact on children’s health, particularly in relation to brain development with regards to the use of essential micronutrients. The data analyzed by the students using ML methods that were shared during the meeting provided strong supporting evidence to individualize dosage regimens for children based on their age in months for the ANRIN project — a national nutrition project supported by the World Bank — as well as policy decisions to extend months of coverage for children, redefining health care practices in Nigeria.

A child receives vitamin A from a community health worker. Photo courtesy M’Care.

MIT students drive change by harnessing the power of data

At the heart of this collaboration lies the contribution of MIT students. Armed with their dedication and skill in data analysis and machine learning, they played a pivotal role in helping M’Care analyze their data and prepare for their meeting with the Ministry of Health. Their most significant findings included ways to identify patients at risk of not completing their full course of micronutrient powder and/or vitamin A, and identifying gaps in M’Care’s data, such as postdated delivery dates and community demographics. These findings are already helping M’Care better plan its resources and adjust the scope of its program to ensure more children complete the intervention.

Darcy Kim, an undergraduate at Wellesley College studying math and computer science, who is cross-registered for the MIT machine learning course, expresses enthusiasm about the practical applications found within the project: “To me, data and math is storytelling, and the story is why I love studying it. … I learned that data exploration involves asking questions about how the data is collected, and that surprising patterns that arise often have a qualitative explanation. Impactful research requires radical collaboration with the people the research intends to help. Otherwise, these qualitative explanations get lost in the numbers.”

Joyce Luo, a first-year operations research PhD student at the Operations Research Center at MIT, shares similar thoughts about the project: “I learned the importance of understanding the context behind data to figure out what kind of analysis might be most impactful. This involves being in frequent contact with the company or organization who provides the data to learn as much as you can about how the data was collected and the people the analysis could help. Stepping back and looking at the bigger picture, rather than just focusing on accuracy or metrics, is extremely important.”

Insights to implementation: A new era for micronutrient dosing

As a direct result of M’Care’s collaboration with MIT, policymakers revamped the dosing scheme for essential micronutrient administration for children in Nigeria to prevent malnutrition. M’Care and MIT’s data analysis unearthed critical insights into the limited frequency of medical visits caused by late-age enrollment. 

“One big takeaway for me was that the data analysis portion of the project — doing a deep dive into the data; understanding, analyzing, visualizing, and summarizing the data — can be just as important as building the machine learning models. M’Care shared our data analysis with the National Ministry of Health, and the insights from it drove them to change their dosing scheme and schedule for delivering micronutrient powder to young children. This really showed us the value of understanding and knowing your data before modeling,” shares Angela Lin, a second-year PhD student at the Operations Research Center.

Armed with this knowledge, policymakers are eager to develop an optimized dosing scheme that caters to the unique needs of children in disadvantaged communities, ensuring maximum impact on their brain development and overall well-being.

Siddharth Srivastava, M’Care’s corporate technology liaison, shares his gratitude for the MIT student’s input. “Collaborating with enthusiastic and driven students was both empowering and inspiring. Each of them brought unique perspectives and technical skills to the table. Their passion for applying machine learning to health care was evident in their unwavering dedication and proactive approach to problem-solving.”

Forging a path to impact

The collaboration between M’Care and MIT exemplifies the remarkable achievements that arise when academia, innovative problem-solvers, and policy authorities unite. By merging academic rigor with real-world expertise, this partnership has the potential to revolutionize child health care not only in Nigeria but also in similar contexts worldwide.

“I believe applying innovative methods of machine learning, data gathering, instrumentation, and planning to real problems in the developing world can be highly effective for those countries and highly motivating for our students. I was happy to have such a project in our class portfolio this year and look forward to future opportunities,” says Peter Szolovits, professor of computer science and engineering at MIT.

By harnessing the power of data, innovation, and collective expertise, this collaboration between M’Care and MIT has the potential to improve equitable child health care in Nigeria. “It has been so fulfilling to see how our team’s work has been able to create even the smallest positive impact in such a short period of time, and it has been amazing to work with a company like Promane and Promade Limited that is so knowledgeable and caring for the communities that they serve,” shares Elizabeth Whittier, a second-year PhD electrical engineering student at MIT.

Planning algorithm enables high-performance flight

Posted on August 23, 2023 by - News

A tailsitter is a fixed-wing aircraft that takes off and lands vertically (it sits on its tail on the landing pad), and then tilts horizontally for forward flight. Faster and more efficient than quadcopter drones, these versatile aircraft can fly over a large area like an airplane but also hover like a helicopter, making them well-suited for tasks like search-and-rescue or parcel delivery.  

The algorithms can execute challenging maneuvers and are so computationally efficient they can plan complex trajectories in real time. The algorithms can execute challenging maneuvers like loops, rolls, and climbing turns, and are so computationally efficient they can plan complex trajectories in real time. Image courtesy of the researchers.

MIT researchers have developed new algorithms for trajectory planning and control of a tailsitter that take advantage of the maneuverability and versatility of this type of aircraft. Their algorithms can execute challenging maneuvers, like sideways or upside-down flight, and are so computationally efficient that they can plan complex trajectories in real-time.

Typically, other methods either simplify the system dynamics in their trajectory planning algorithm or use two different models, one for helicopter mode and one for airplane mode. Neither approach can plan and execute trajectories that are as aggressive as those demonstrated by the MIT team.

“We wanted to really exploit all the power the system has. These aircraft, even if they are very small, are quite powerful and capable of exciting acrobatic maneuvers. With our approach, using one model, we can cover the entire flight envelope — all the conditions in which the vehicle can fly,” says Ezra Tal, a research scientist in the Laboratory for Information and Decision Systems (LIDS) and lead author of a new paper describing the work.

Tal and his collaborators used their trajectory generation and control algorithms to demonstrate tailsitters that perform complex maneuvers like loops, rolls, and climbing turns, and they even showcased a drone race where three tailsitters sped through aerial gates and performed several synchronized, acrobatic maneuvers.

MIT researchers developed trajectory generation and control algorithms that enable a tailsitter, a type of highly maneuverable fixed-wing aircraft shown here, to perform complex maneuvers like spins, loops, and climbing turns. Image courtesy of the researchers.

These algorithms could potentially enable tailsitters to autonomously perform complex moves in dynamic environments, such as flying into a collapsed building and avoiding obstacles while on a rapid search for survivors.

Joining Tal on the paper are Gilhyun Ryou, a graduate student in the Department of Electrical Engineering and Computer Science (EECS); and senior author Sertac Karaman, associate professor of aeronautics and astronautics and director of LIDS. The research appears in IEEE Transactions on Robotics.

Tackling tailsitter trajectories

The design for a tailsitter was invented by Nikolai Tesla in 1928, but no one tried to seriously build one until nearly 20 years after his patent was filed. Even today, due to the complexity of tailsitter motion, research and commercial applications have tended to focus on aircraft that are easier to control, like quadcopter drones.

Trajectory generation and control algorithms that do exist for tailsitters mostly focus on calm trajectories and slow transitions, rather than the rapid and acrobatic maneuvers these aircraft are capable of making.

With such challenging flight conditions, Tal and his collaborators knew they would need to design trajectory planning and control algorithms specifically for agile trajectories with fast-changing accelerations in order to enable these unique aircraft to reach peak performance.

To do that, they used a global dynamics model, meaning one that applies to all flight conditions, ranging from vertical take-off to forward, or even sideways, flight. Next, they leveraged a technical property known as differential flatness to ensure that model would perform efficiently.

In trajectory generation, a key step is to ensure the aircraft can actually fly the planned trajectory — maybe it has a minimum turning radius that makes a particularly sharp corner infeasible. Since tailsitters are complex systems, with flaps and rotors, and exhibit such complicated aerial motions, it typically takes numerous calculations to determine if a trajectory is feasible, which hampers traditional planning algorithms.

By employing differential flatness, the MIT researchers can use a mathematical function to quickly check whether a trajectory is feasible. Their approach avoids many of the complicated system dynamics and plans a trajectory for the tailsitter as a mathematical curve through space. The algorithm then uses differential flatness to rapidly check the feasibility of that trajectory.

“That check is computationally very cheap, so that is why with our algorithm, you can actually plan trajectories in real-time,” Tal explains.

These trajectories can be very complex, rapidly transitioning between vertical and horizontal flight while incorporating sideways and inverted maneuvers, because the researchers designed their algorithm in such a way that it uniformly considers all of these diverse flight conditions.

“Many research teams focused on the quadcopter aircraft, which is very common configuration for almost all consumer drones. The tailsitters, on the other hand, are a lot more efficient in forward flight. I think they were not used as much because they are much harder to pilot,” Karaman says. “But, the kind of autonomy technology we developed suddenly makes them available in many applications, from consumer technology to large-scale industrial inspections.”

A tailsitter airshow

They put their method to the test by planning and executing a number of challenging trajectories for tailsitters in MIT’s indoor flight space. In one test, they demonstrate a tailsitter executing a climbing turn where the aircraft turns to the left and then rapidly accelerates and banks back to the right.

They also showcased a tailsitter “airshow” in which three synchronized tailsitters performed loops, sharp turns, and flew seamlessly through airborne gates. These maneuvers wouldn’t be possible to plan in real-time without their model’s use of differential flatness, says Tal.

“Differential flatness was developed and applied to generate smooth trajectories for basic mechanical systems, such as a motorized pendulum. Now, more than 30 years later, we’ve applied it to fixed-wing aircraft. There might be many other applications we could apply this to in the future,” Ryou adds.

The next step for the MIT researchers is to extend their algorithm so it could be used effectively for fully autonomous outdoor flight, where winds and other environmental conditions can drastically affect the dynamics of a fixed-wing aircraft.

This work was supported, in part, by the U.S. Army Research Office.

Machine-learning system based on light could yield more powerful, efficient large language models

Posted on August 23, 2023 by - News

ChatGPT has made headlines around the world with its ability to write essays, email, and computer code based on a few prompts from a user. Now an MIT-led team reports a system that could lead to machine-learning programs several orders of magnitude more powerful than the one behind ChatGPT. The system they developed could also use several orders of magnitude less energy than the state-of-the-art supercomputers behind the machine-learning models of today.

In the July 17 issue of Nature Photonics, the researchers report the first experimental demonstration of the new system, which performs its computations based on the movement of light, rather than electrons, using hundreds of micron-scale lasers. With the new system, the team reports a greater than 100-fold improvement in energy efficiency and a 25-fold improvement in compute density, a measure of the power of a system, over state-of-the-art digital computers for machine learning. 

Toward the future

In the paper, the team also cites “substantially several more orders of magnitude for future improvement.” As a result, the authors continue, the technique “opens an avenue to large-scale optoelectronic processors to accelerate machine-learning tasks from data centers to decentralized edge devices.” In other words, cellphones and other small devices could become capable of running programs that can currently only be computed at large data centers.

Further, because the components of the system can be created using fabrication processes already in use today, “we expect that it could be scaled for commercial use in a few years. For example, the laser arrays involved are widely used in cell-phone face ID and data communication,” says Zaijun Chen, first author, who conducted the work while a postdoc at MIT in the Research Laboratory of Electronics (RLE) and is now an assistant professor at the University of Southern California.

Says Dirk Englund, an associate professor in MIT’s Department of Electrical Engineering and Computer Science and leader of the work, “ChatGPT is limited in its size by the power of today’s supercomputers. It’s just not economically viable to train models that are much bigger. Our new technology could make it possible to leapfrog to machine-learning models that otherwise would not be reachable in the near future.”

He continues, “We don’t know what capabilities the next-generation ChatGPT will have if it is 100 times more powerful, but that’s the regime of discovery that this kind of technology can allow.” Englund is also leader of MIT’s Quantum Photonics Laboratory and is affiliated with the RLE and the Materials Research Laboratory.

A drumbeat of progress

The current work is the latest achievement in a drumbeat of progress over the last few years by Englund and many of the same colleagues. For example, in 2019 an Englund team reported the theoretical work that led to the current demonstration. The first author of that paper, Ryan Hamerly, now of RLE and NTT Research Inc., is also an author of the current paper.

Additional coauthors of the current Nature Photonics paper are Alexander Sludds, Ronald Davis, Ian Christen, Liane Bernstein, and Lamia Ateshian, all of RLE; and Tobias Heuser, Niels Heermeier, James A. Lott, and Stephan Reitzensttein of Technische Universitat Berlin.

Deep neural networks (DNNs) like the one behind ChatGPT are based on huge machine-learning models that simulate how the brain processes information. However, the digital technologies behind today’s DNNs are reaching their limits even as the field of machine learning is growing. Further, they require huge amounts of energy and are largely confined to large data centers. That is motivating the development of new computing paradigms.

Using light rather than electrons to run DNN computations has the potential to break through the current bottlenecks. Computations using optics, for example, have the potential to use far less energy than those based on electronics. Further, with optics, “you can have much larger bandwidths,” or compute densities, says Chen. Light can transfer much more information over a much smaller area.

But current optical neural networks (ONNs) have significant challenges. For example, they use a great deal of energy because they are inefficient at converting incoming data based on electrical energy into light. Further, the components involved are bulky and take up significant space. And while ONNs are quite good at linear calculations like adding, they are not great at nonlinear calculations like multiplication and “if” statements.

In the current work the researchers introduce a compact architecture that, for the first time, solves all of these challenges and two more simultaneously. That architecture is based on state-of-the-art arrays of vertical surface-emitting lasers (VCSELs), a relatively new technology used in applications including lidar remote sensing and laser printing. The particular VCELs reported in the Nature Photonics paper were developed by the Reitzenstein group at Technische Universitat Berlin. “This was a collaborative project that would not have been possible without them,” Hamerly says.

Logan Wright, an assistant professor at Yale University who was not involved in the current research, comments, “The work by Zaijun Chen et al. is inspiring, encouraging me and likely many other researchers in this area that systems based on modulated VCSEL arrays could be a viable route to large-scale, high-speed optical neural networks. Of course, the state of the art here is still far from the scale and cost that would be necessary for practically useful devices, but I am optimistic about what can be realized in the next few years, especially given the potential these systems have to accelerate the very large-scale, very expensive AI systems like those used in popular textual ‘GPT’ systems like ChatGPT.”

Chen, Hamerly, and Englund have filed for a patent on the work, which was sponsored by the U.S. Army Research Office, NTT Research, the U.S. National Defense Science and Engineering Graduate Fellowship Program, the U.S. National Science Foundation, the Natural Sciences and Engineering Research Council of Canada, and the Volkswagen Foundation.

MIT Code for Good Club works with local nonprofits

Posted on August 17, 2023 by - News

Computer hackers who break into websites, change the code, and do harm are very real. But MIT Code for Good members want to do just the opposite. This group of mostly electrical engineering and computer science majors (EECS, Course 6) wants to help important causes. Each semester, club members consult with nonprofits in the Boston area to support their technical needs.

Formed in 2016, the club currently has 20 undergraduate members, but graduate students are also welcome. The work varies from project to project but often involves website and blog creation, software support, app development, and data analysis and visualization. Past clients include Parents Helping Parents, Chinese Cultural Connection, The Depression and Bipolar Support Alliance of Boston, and the Boston Tax Help Coalition, among others.

“We reach out to local nonprofits and ask if there are any projects that we can help them with, or if there are any projects that they want to do that they do not have the technical ability to do,” says Will Reed, club officer. “There are no course requirements for students to join the club, and students with any background or major are encouraged to apply. Students who are not computer science majors will be put on a team or project that needs their skill set. The time commitment is three to five hours a week.”

“It’s great to be able to collaborate on a project with a team and to be able to build something that is able to impact others through the nonprofits we serve. Our projects help us to learn and grow as developers, while also making something meaningful and purposeful,” says Isaac Taylor, a Course 6 junior.

“A class I found especially helpful is 6.s063 (Design for the Web: Languages and User Interfaces), a new developing class that teaches web development through the lens of usability and design principles. It provides a good foundation in many of the skills applicable to the projects in Code for Good. The coding-focused classes — especially 6.100A/B, 6.101, 6.102 — are also helpful to get more comfortable with software development,” says Taylor.

“The organizations that we work with appreciate the work we do, and nonprofits regularly decide to continue to work with the club,” says Reed.

Students interested in such work should keep an eye out this fall for the application if they’re interested in becoming a club member. The club will also be recruiting for executive positions. For more information visit the website. Organizations in the Boston area interested in collaborating with the club can contact them via email.

A faster way to teach a robot

Posted on August 10, 2023 by - News

Imagine purchasing a robot to perform household tasks. This robot was built and trained in a factory on a certain set of tasks and has never seen the items in your home. When you ask it to pick up a mug from your kitchen table, it might not recognize your mug (perhaps because this mug is painted with an unusual image, say, of MIT’s mascot, Tim the Beaver). So, the robot fails.

“Right now, the way we train these robots, when they fail, we don’t really know why. So you would just throw up your hands and say, ‘OK, I guess we have to start over.’ A critical component that is missing from this system is enabling the robot to demonstrate why it is failing so the user can give it feedback,” says Andi Peng, an electrical engineering and computer science (EECS) graduate student at MIT.

Peng and her collaborators at MIT, New York University, and the University of California at Berkeley created a framework that enables humans to quickly teach a robot what they want it to do, with a minimal amount of effort.

When a robot fails, the system uses an algorithm to generate counterfactual explanations that describe what needed to change for the robot to succeed. For instance, maybe the robot would have been able to pick up the mug if the mug were a certain color. It shows these counterfactuals to the human and asks for feedback on why the robot failed. Then the system utilizes this feedback and the counterfactual explanations to generate new data it uses to fine-tune the robot.

Fine-tuning involves tweaking a machine-learning model that has already been trained to perform one task, so it can perform a second, similar task.

The researchers tested this technique in simulations and found that it could teach a robot more efficiently than other methods. The robots trained with this framework performed better, while the training process consumed less of a human’s time.

This framework could help robots learn faster in new environments without requiring a user to have technical knowledge. In the long run, this could be a step toward enabling general-purpose robots to efficiently perform daily tasks for the elderly or individuals with disabilities in a variety of settings.

Peng, the lead author, is joined by co-authors Aviv Netanyahu, an EECS graduate student; Mark Ho, an assistant professor at the Stevens Institute of Technology; Tianmin Shu, an MIT postdoc; Andreea Bobu, a graduate student at UC Berkeley; and senior authors Julie Shah, an MIT professor of aeronautics and astronautics and the director of the Interactive Robotics Group in the Computer Science and Artificial Intelligence Laboratory (CSAIL), and Pulkit Agrawal, an EECS professor and CSAIL affiliate. The research will be presented at the International Conference on Machine Learning.

On-the-job training

Robots often fail due to distribution shift — the robot is presented with objects and spaces it did not see during training, and it doesn’t understand what to do in this new environment.

One way to retrain a robot for a specific task is imitation learning. The user could demonstrate the correct task to teach the robot what to do. If a user tries to teach a robot to pick up a mug, but demonstrates with a white mug, the robot could learn that all mugs are white. It may then fail to pick up a red, blue, or “Tim-the-Beaver-brown” mug.

Training a robot to recognize that a mug is a mug, regardless of its color, could take thousands of demonstrations.

“I don’t want to have to demonstrate with 30,000 mugs. I want to demonstrate with just one mug. But then I need to teach the robot so it recognizes that it can pick up a mug of any color,” Peng says.

To accomplish this, the researchers’ system determines what specific object the user cares about (a mug) and what elements aren’t important for the task (perhaps the color of the mug doesn’t matter). It uses this information to generate new, synthetic data by changing these “unimportant” visual concepts. This process is known as data augmentation.

The framework has three steps. First, it shows the task that caused the robot to fail. Then it collects a demonstration from the user of the desired actions and generates counterfactuals by searching over all features in the space that show what needed to change for the robot to succeed.

The system shows these counterfactuals to the user and asks for feedback to determine which visual concepts do not impact the desired action. Then it uses this human feedback to generate many new augmented demonstrations.

In this way, the user could demonstrate picking up one mug, but the system would produce demonstrations showing the desired action with thousands of different mugs by altering the color. It uses these data to fine-tune the robot.

Creating counterfactual explanations and soliciting feedback from the user are critical for the technique to succeed, Peng says.

From human reasoning to robot reasoning

Because their work seeks to put the human in the training loop, the researchers tested their technique with human users. They first conducted a study in which they asked people if counterfactual explanations helped them identify elements that could be changed without affecting the task.

“It was so clear right off the bat. Humans are so good at this type of counterfactual reasoning. And this counterfactual step is what allows human reasoning to be translated into robot reasoning in a way that makes sense,” she says.

Then they applied their framework to three simulations where robots were tasked with: navigating to a goal object, picking up a key and unlocking a door, and picking up a desired object then placing it on a tabletop. In each instance, their method enabled the robot to learn faster than with other techniques, while requiring fewer demonstrations from users.

Moving forward, the researchers hope to test this framework on real robots. They also want to focus on reducing the time it takes the system to create new data using generative machine-learning models.

“We want robots to do what humans do, and we want them to do it in a semantically meaningful way. Humans tend to operate in this abstract space, where they don’t think about every single property in an image. At the end of the day, this is really about enabling a robot to learn a good, human-like representation at an abstract level,” Peng says.

This research is supported, in part, by a National Science Foundation Graduate Research Fellowship, Open Philanthropy, an Apple AI/ML Fellowship, Hyundai Motor Corporation, the MIT-IBM Watson AI Lab, and the National Science Foundation Institute for Artificial Intelligence and Fundamental Interactions.

Two professors awarded 2023 Thornton Faculty Research Innovation Fellowships (FRIFs)

Posted on August 10, 2023 by - EECS Celebrates Awards

Professor Adam Chlipala and Professor of EECS and in the Institute for Data, Systems and Society (IDSS) Caroline Uhler have been awarded the 2023 Thornton Faculty Research Innovation Fellowship (FRIF) awards. Funded by a generous donation from the late Professor Emeritus Richard “Dick” Thornton SM ’54, ScD ’57, the FRIF awards were established in 2011 to provide tenured, mid-career faculty with the funding resources and freedom necessary to explore new research directions, resulting in potentially important discoveries through early-stage research.

Adam Chlipala joined MIT in 2011 and is a Professor of EECS. He earned his BS from Carnegie Mellon University (CMU) in 2003, and his MS and PhD from Berkeley in 2004 and 2007, respectively. He spent time at Jane Street as a software developer and Harvard as a postdoc before joining MIT. Chlipala is the head of the Programming Languages and Verification Group in CSAIL, where his research focuses on developing methods for integrating the work of software design and verification. He has also done extensive foundational work in building general computational infrastructure to support programming, verification, and automatic code generation. Chlipala has applied his techniques to several key systems areas such as file system verification, hardware design, and cryptographic libraries for use in building secure systems. His recent work on cryptographic libraries has been adopted by Google for its Chrome web browser, and his formal semantics for the RISC-V processor have recently been adopted as the official specification for the processor’s instruction set architecture. 

Among other honors, Chlipala has been awarded a 2013 NSF CAREER award, a Best Paper award at SOSP 2015 for his FSCQ work on file system verification, the Most Influential Paper award at the International Conference on Functional Programming (ICFP) 2018 and two Communication of the ACM (CACM) research highlights. He was also elected as ACM Distinguished Member in 2019.

Caroline Uhler joined the MIT faculty in 2015 and is currently a full professor in EECS (Electrical Engineering & Computer Science) and IDSS (Institute for Data, Systems and Society). At MIT, she is also affiliated with LIDS (Laboratory for Information and Decision Systems), the Center for Statistics, Machine Learning at MIT, and the ORC (Operations Research Center). In addition, she is a core member of the Broad Institute of MIT and Harvard, where she is a co-director of the Eric and Wendy Schmidt Center. Uhler’s research focuses on machine learning, statistics and computational biology, in particular on causal inference, generative modeling, and applications to genomics. Her use of probabilistic graphical models and development of scalable algorithms with healthcare applications has enabled her research group to gain insights into causal relationships hidden within massive amounts of data (such as those generated during gene knockout or knockdown experiments.) 

Uhler holds an MSc in mathematics, a BSc in biology, and an MEd in mathematics education from the University of Zurich, and a PhD in statistics from UC Berkeley. Before joining MIT, she spent a semester in the “Big Data” program at the Simons Institute at UC Berkeley, held postdoctoral positions at the IMA and at ETH Zurich, and spent 3 years as an assistant professor at IST Austria. She is an elected member of the International Statistical Institute, and is the recipient of a Simons Investigator Award, a Sloan Research Fellowship, an NSF Career Award, a Sofja Kovalevskaja Award from the Humboldt Foundation, and a START Award from the Austrian Science Foundation. Recently, she was named a Fellow of the Society for Industrial and Applied Mathematics (SIAM), Class of 2023. 

Celebrating the impact of IDSS

Posted on August 10, 2023 by - News

The “interdisciplinary approach” is something that has been lauded for decades for its ability to break down silos and create new integrated approaches to research.

For Munther Dahleh, founding director of the MIT Institute for Data, Systems, and Society (IDSS), showing the community that data science and statistics can transcend individual disciplines and form a new holistic approach to addressing complex societal challenges has been crucial to the institute’s success.

“From the very beginning, it was critical that we recognized the areas of data science, statistics, AI, and, in a way, computing, as transdisciplinary,” says Dahleh, who is the William A. Coolidge Professor in Electrical Engineering and Computer Science. “We made that point over and over — these are areas that embed in your field. It is not ours; this organization is here for everyone.”

On April 14-15, researchers from across and beyond MIT joined together to celebrate the accomplishments and impact IDSS has had on research and education since its inception in 2015. Taking the place of IDSS’s annual statistics and data science conference SDSCon, the celebration also doubled as a way to recognize Dahleh for his work creating and executing the vision of IDSS as he prepares to step down from his director position this summer.

In addition to talks and panels on statistics and computation, smart systems, automation and artificial intelligence, conference participants discussed issues ranging from climate change, health care, and misinformation. Nobel Prize winner and IDSS affiliate Professor Esther Duflo spoke on large scale immunization efforts, former MLK Visiting Professor Craig Watkins joined a panel on equity and justice in AI, and IDSS Associate Director Alberto Abadie discussed synthetic controls for policy evaluation. Other policy questions were explored through lightning talks, including those by students from the Technology and Policy Program (TPP) within IDSS.

A place to call home

The list of IDSS accomplishments over the last eight years is long and growing. From creating a home for 21st century statistics at MIT after other unsuccessful attempts, to creating a new PhD preparing the trilingual student who is an expert in data science and social science in the context of a domain, to playing a key role in determining an effective process for Covid testing in the early days of the pandemic, IDSS has left its mark on MIT. More recently, IDSS launched an initiative using big data to help effect structural and normative change toward racial equity, and will continue to explore societal challenges through the lenses of statistics, social science, and science and engineering.

“I’m very proud of what we’ve done and of all the people who have contributed to this. The leadership team has been phenomenal in their commitment and their creativity,” Dahleh says. “I always say it doesn’t take one person, it takes the village to do what we have done, and I am very proud of that.”

Prior to the institute’s formation, Dahleh and others at MIT were brought together to answer one key question: How would MIT prepare for the future of systems and data?

“Data science is a complex area because in some ways it’s everywhere and it belongs to everyone, similar to statistics and AI,” Dahleh says “The most important part of creating an organization to support it was making it clear that it was an organization for everyone.” The response the team came back with was to build an Institute: a department that could cross all other departments and schools.

While Dahleh and others on the committee were creating this blueprint for the future, the events that would lead early IDSS hires like Caroline Uhler to join the team were also beginning to take shape. Uhler, now an MIT professor of computer science and co-director of the Eric and Wendy Schmidt Center at the Broad Institute, was a panelist at the celebration discussing statistics and human health.

In 2015, Uhler was a faculty member at the Institute of Science and Technology in Austria looking to move back to the U.S. “I was looking for positions in all different types of departments related to statistics, including electrical engineering and computer science, which were areas not related to my degree,” Uhler says. “What really got me to MIT was Munther’s vision for building a modern type of statistics, and the unique opportunity to be part of building what statistics should be moving forward.”

The breadth of the Statistics and Data Science Center has given it a unique and a robust character that makes for an attractive collaborative environment at MIT. “A lot of IDSS’s impact has been in giving people like me a home,” Uhler adds. “By building an institute for statistics that is across all schools instead of housed within a single department, it has created a home for everyone who is interested in the field.”

Filling the gap

For Ali Jadbabaie, former IDSS associate director and another early IDSS hire, being in the right place at the right time landed him in the center of it all. A control theory expert and network scientist by training, Jadbabaie first came to MIT during a sabbatical from his position as a professor at the University of Pennsylvania.

“My time at MIT coincided with the early discussions around forming IDSS and given my experience they asked me to stay and help with its creation,” Jadbabaie says. He is now head of the Department of Civil and Environmental Engineering at MIT, and he spoke at the celebration about a new MIT major in climate system science and engineering.

A critical early accomplishment of IDSS was the creation of a doctoral program in social and engineering systems (SES), which has the goal of educating and fostering the success of a new type of PhD student, says Jadbabaie.

“We realized we had this opportunity to educate a new type of PhD student who was conversant in the math of information sciences and statistics in addition to an understanding of a domain — infrastructures, climate, political polarization — in which problems arise,” he says. “This program would provide training in statistics and data science, the math of information sciences and a branch of social science that is relevant to their domain.”

“SES has been filling a gap,” adds Jadbabaie. “We wanted to bring quantitative reasoning to areas in social sciences, particularly as they interact with complex engineering systems.”

“My first year at MIT really broadened my horizon in terms of what was available and exciting,” says Manxi Wu, a member of the first cohort of students in the SES program after starting out in the Master of Science in Transportation (MST) program. “My advisor introduced me to a number of interesting topics at the intersection of game theory, economics, and engineering systems, and in my second year I realized my interest was really about the societal scale systems, with transportation as my go-to application area when I think about how to make an impact in the real world.”

Wu, now an assistant professor in the School of Operations Research and Information Engineering at Cornell, was a panelist at the Celebration’s session on smart infrastructure systems. She says that the beauty of the SES program lies in its ability to create a common ground between groups of students and researchers who all have different applications interests but share an eagerness to sharpen their technical skills.

“While we may be working on very different application areas, the core methodologies, such as mathematical tools for data science and probability optimization, create a common language,” Wu says. “We are all capable of speaking the technical language, and our diversified interests give us even more to talk about.”

In addition to the PhD program, IDSS has helped bring quality MIT programming to people around the globe with its MicroMasters Program in Statistics and Data Science (SDS), which recently celebrated the certification of over 1,000 learners. The MicroMasters is just one offering in the newly-minted IDSSx, a collection of online learning opportunities for learners at different skill levels and interests.

“The impact of branding what MIT-IDSS does across the globe has been great,” Dahleh says. “In addition, we’ve created smaller online programs for continued education in data science and machine learning, which I think is also critical in educating the community at large.”

Hopes for the future

Through all of its accomplishments, the core mission of IDSS has never changed.

“The belief was always to create an institute focused on how data science can be used to solve pressing societal problems,” Dahleh says. “The organizational structure of IDSS as an MIT Institute has enabled it to promote data and systems as a transdiciplinary area that embeds in every domain to support its mission. This reverse ownership structure will continue to strengthen the presence of IDSS in MIT and will make it an essential unit within the Schwarzman College of Computing.”

As Dahleh prepares to step down from his role, and Professor Martin Wainwright gets ready to fill his (very big) shoes as director, Dahleh’s colleagues say the real key to the success of IDSS all started with his passion and vision.

“Creating a new academic unit within MIT is actually next to impossible,” Jadbabaie says. “It requires structural changes, as well as someone who has a strong understanding of multiple areas, who knows how to get people to work together collectively, and who has a mission.”

“The most important thing is that he was inclusive,” he adds. “He didn’t try to create a gate around it and say these people are in and these people are not. I don’t think this would have ever happened without Munther at the helm.”

New sensor mimics cell membrane functions

Posted on August 9, 2023 by - News

Drawing inspiration from natural sensory systems, an MIT-led team has designed a novel sensor that could detect the same molecules that naturally occurring cell receptors can identify.

In work that combines several new technologies, the researchers created a prototype sensor that can detect an immune molecule called CXCL12, down to tens or hundreds of parts per billion. This is an important first step to developing a system that could be used to perform routine screens for hard-to-diagnose cancers or metastatic tumors, or as a highly biomimetic electronic “nose,” the researchers say.

“Our hope is to develop a simple device that lets you do at-home testing, with high specificity and sensitivity. The earlier you detect cancer, the better the treatment, so early diagnostics for cancer is one important area we want to go in,” says Shuguang Zhang, a principal research scientist in MIT’s Media Lab.

The device draws inspiration from the membrane that surrounds all cells. Within such membranes are thousands of receptor proteins that detect molecules in the environment. The MIT team modified some of these proteins so that they could survive outside the membrane, and anchored them in a layer of crystallized proteins atop an array of graphene transistors. When the target molecule is detected in a sample, these transistors relay the information to a computer or smartphone.

This type of sensor could potentially be adapted to analyze any bodily fluid, such as blood, tears, or saliva, the researchers say, and could screen for many different targets simultaneously, depending on the type of receptor proteins used.

“We identify critical receptors from biological systems and anchor them onto a bioelectronic interface, allowing us to harvest all those biological signals and then transduce them into electrical outputs that can be analyzed and interpreted by machine-learning algorithms,” says Rui Qing, a former MIT research scientist who is now an associate professor at Shanghai Jiao Tong University.

Qing and Mantian Xue PhD ’23, are the lead authors of the study, which appears today in Science Advances. Along with Zhang, Tomás Palacios, director of MIT’s Microsystems Laboratory and a professor of electrical engineering and computer science, and Uwe Sleytr, an emeritus professor at the Institute of Synthetic Bioarchitectures at the University of Natural Resources and Life Sciences in Vienna, are senior authors of the paper.

“What we are aiming to do is develop the basic technology to enable a future portable device that we can integrate with cell phones and computers, so that you can do a test at home and quickly find out whether you should go to the doctor,” Qing says. Image courtesy of the researchers.

Free from membranes

Most current diagnostic sensors are based on either antibodies or aptamers (short strands of DNA or RNA) that can capture a particular target molecule from a fluid such as blood. However, both of these approaches have limitations: Aptamers can be easily broken down by body fluids, and manufacturing antibodies so that every batch is identical can be difficult.

One alternative approach that scientists have explored is building sensors based on the receptor proteins found in cell membranes, which cells use to monitor and respond to their environment. The human genome encodes thousands of such receptors. However, these receptor proteins are difficult to work with because once removed from the cell membrane, they only maintain their structure if they are suspended in a detergent.

In 2018, Zhang, Qing, and others reported a novel way to transform hydrophobic proteins into water-soluble proteins, by swapping out a few hydrophobic amino acids for hydrophilic amino acids. This approach is called the QTY code, after the letters representing the three hydrophilic amino acids — glutamine, threonine, and tyrosine — that take the place of hydrophobic amino acids leucine, isoleucine, valine, and phenylalanine.  

“People have tried to use receptors for sensing for decades, but it is challenging for widespread use because receptors need detergent to keep them stable. The novelty of our approach is that we can make them water-soluble and can produce them in large quantities, inexpensively,” Zhang says.

Zhang and Sleytr, who are longtime collaborators, decided to team up to try to attach water-soluble versions of receptor proteins to a surface, using bacterial proteins that Sleytr has studied for many years. These proteins, known as S-layer proteins, are found as the outermost surface layer of the cell envelope in many types of bacteria and archaea.

When S-layer proteins are crystallized, they form coherent monomolecular arrays on a surface. Sleytr had previously shown that these proteins can be fused with other proteins such as antibodies or enzymes. For this study, the researchers, including senior scientist Andreas Breitwieser, who is also a co-author in the paper, used S-layer proteins to create a very dense, immobilized sheet of a water-soluble version of a receptor protein called CXCR4. This receptor binds to a target molecule called CXCL12, which plays important roles in several human diseases including cancer, and to an HIV coat glycoprotein, which is responsible for virus entry into human cells.

“We use these S-layer systems to allow all these functional molecules to attach to a surface in a monomolecular array, in a very well-defined distribution and orientation,” Sleytr says. “It’s like a chessboard where you can arrange different pieces in a very precise manner.”

The researchers named their sensing technology RESENSA (Receptor S-layer Electrical Nano Sensing Array).

Sensitivity with biomimicry

These crystallized S-layers can be deposited onto nearly any surface. For this application, the researchers attached the S-layer to a chip with graphene-based transistor arrays that Palacios’ lab had previously developed. The single-atomic thickness of the graphene transistors makes them ideal for the development of highly sensitive detectors.

Working in Palacios’ lab, Xue adapted the chip so that it could be coated with a dual layer of proteins — crystallized S-layer proteins attached to water-soluble receptor proteins. When a target molecule from the sample binds to a receptor protein, the charge of the target changes the electrical properties of the graphene in a way that can be easily quantified and transmitted to a computer or smartphone connected to the chip.

“We chose graphene as the transducer material because it has excellent electrical properties, meaning it can better translate those signals. It has the highest surface-to-volume ratio because it’s a sheet of carbon atoms, so every change on the surface, caused by the protein binding events, translates directly to the whole bulk of the material,” Xue says.

The graphene transistor chip can be coated with S-layer-receptor proteins with a density of 1 trillion receptors per square centimeter with upward orientation. This allows the chip to take advantage of the maximum sensitivity offered by the receptor proteins, within the clinically relevant range for target analytes in human bodies. The array chip integrates more than 200 devices, providing a redundancy in signal detection that helps to ensure reliable measurements even in the case of rare molecules, such as the ones that could reveal the presence of an early-stage tumor or the onset of Alzheimer’s disease, the researchers say.

Thanks to the use of QTY code, it is possible to modify naturally existing receptor proteins that could then be used, the researchers say, to generate an array of sensors in a single chip to screen virtually any molecule that cells can detect. “What we are aiming to do is develop the basic technology to enable a future portable device that we can integrate with cell phones and computers, so that you can do a test at home and quickly find out whether you should go to the doctor,” Qing says.

“This new system is the combination of different research fields as molecular and synthetic biology, physics, and electrical engineering, which in the team’s approach are nicely integrated,” says Piero Baglioni, a professor of physical chemistry at the University of Florence, who was not involved in the study. “Moreover, I believe that it is a breakthrough that could be very useful in diagnostics of many diseases.”

The research was funded by the National Science Foundation, MIT Institute for Soldier Nanotechnologies, and Wilson Chu of Defond Co. Ltd.

Four MIT students selected for Qualcomm Innovation Fellowship 2023

Posted on August 9, 2023 by - EECS Celebrates Awards

Two PhD students from the MIT Media Lab’s Camera Culture research group, Tzofi Klinghoffer and Kushagra Tiwary, are among the winners of the 2023 Qualcomm Innovation Fellowship—for the North America Program—based on their joint proposal, “Enabling Novel XR Experiences with Time-of-flight Sensing.” 

Hae Won Lee and Jiadi Zhu, PhD students in the MIT Department of Electrical Engineering and Computer Science (EECS), have also been chosen as winners of the fellowship this year, for their proposal titled “Low-temperature Heterogeous Integration of MoS2 Transistors on Silicon CMOS Circuits for RF Energy Harvesting”. The selection process included detailed research proposals and presentations, and 18 teams were selected out of 182 submissions.  

The Qualcomm Innovation Fellowship (QIF) is focused on recognizing, rewarding, and mentoring innovative PhD students across a broad range of technical research areas. QIF enables graduate students to be mentored by Qualcomm Research’s top engineers and supports them in their quest towards achieving their research goals. The QIF receives over 100 proposals each year in the US and internationally combined, and has awarded over $15 million dollars since it started in 2009 at Qualcomm’s Research Center in Silicon Valley, California.