The emerging Generative AI landscape
Artificial Intelligence (AI) has been a hot technology topic for the past decade, but generative AI, and specifically the arrival of ChatGPT in 2022, has thrust AI into the spotlight globally and has launched an unprecedented surge of AI innovation and adoption. Generative AI, sometimes referred to GenAI, is AI that can create new content such as text, images, video, audio, or software code in response to a user’s prompt/request. Generative AI systems are built upon a limited number of foundation models, i.e., very large deep learning models trained on vast amounts of training datasets to perform multiple tasks (IBM, What is generative AI?). Generative AI is being applied widely across different industries, including education, entertainment, healthcare, scientific research, and cybersecurity. This paper outlines some of the key issues and considerations relating to generative AI.
Generative AI systems have been developed across multiple modalities, with the earliest and most notable progress achieved in text generation. Well-known GenAI models for text include GPT-4 and LLAMA 3. Capable generative AI systems for images have emerged, including DALLE-3 and Stable Diffusion. The latest generative AI systems, GPT-4 or 4M-21[i], are trained on multimodal input, i.e., text and image, and can generate output in different modalities.
According to the Artificial Intelligence Index Report 2024, AI continues to make advancements that see it outperform humans on certain tasks such as image classification as well as help certain workers be more productive and accelerate scientific research. This AI charge is being led by industry rather than academia and the money to invest in these ever more expensive machine learning models is flowing into the former much more so than the latter.
According to AI Index estimates, the training costs of state-of-the-art AI models have reached unprecedented levels. For example, OpenAI’s GPT-4 used an estimated $78 million worth of computing power to train, while Google’s Gemini Ultra cost $191 million for computing power. These costs are prohibitively expensive meaning very few entities can operate at that level of expense leading to a concentration at the top of the AI provider pyramid. On a regional level, the United States is the leading source of top AI models. In 2023, 61 notable AI models originated from U.S.-based institutions, far outpacing the European Union’s 21 and China’s 15.
Generative AI is gaining popularity as a means of creating autonomous AI agents[ii],[iii] that can process multiple input sources, interact with tools or other generative AI systems, and make automated decisions. These agents rely on generative AI systems to serve as their brains to coordinate tasks and make inferences. Adopting generative AI in AI agents increases its impact enormously while simultaneously amplifying its serious vulnerabilities. One major issue is that current generative AI systems are not built to be secure-by-design, making them vulnerable to exploitation by malicious actors who can easily circumvent their safeguards (e.g. by prompt injection attacks).
Complementing this AI charge are the increasing number of regulations being published to address citizens’ growing concerns over the role of AI in society. However, rigorous, standardised evaluations of these ever-prevalent AI models are seriously lacking. The leading developers are free to conduct responsible AI benchmarking as they see fit which complicates efforts to systematically compare the risks and limitations of top AI models. According to the UN High Commissioner for Human Rights, AI has the potential to benefit humanity, but regulation is essential to ensure these benefits outweigh the risks. Regulations should be based on human rights principles, requiring transparency, independent oversight, and effective remedies (Artificial intelligence must be grounded in human rights, says High Commissioner)
Impact on education
Generative AI has huge potential to affect our daily lives and society in general, including the education sector. Generative AI can make teachers more effective when preparing lessons and students can receive more personalised learning experiences. More administrative tasks can be automated, which enables teachers to spend more time interacting with students. Moreover, generative AI can make individualised learning possible by adapting to the student’s style and pace of study. Instant feedback increases learning efficiency, and accessibility is enhanced through translation or in other formats for learners with very different learning abilities. These advantages provide a quality improvement in the delivery of education.
According to Shaping the Future of Learning: The Role of AI in Education 4.0 from the World Economic Forum, the West African ed-tech start-up, Kabakoo Academies, utilises a Large Language Model (LLM) for the creation of GenAI-powered virtual mentors to guide young people in developing the mindset and skills needed for self-employment. The above programme yields several positive effects:
- Personalised Learning Support: LLM-based AI mentor is available 24/7 for advice, access to information, and customised feedback for better learner support.
- Accessibility of Language: Development of an AI model for education in Bambara, the most widely spoken language in Mali, improves educational accessibility for those with diverse linguistic backgrounds.
- Tangible Outcomes: This programme led to a 23% shift in learners’ growth mindset and incomes increased by 44% six months after completion.
At the same time, generative AI poses several challenges. One major concern is the reliability of the content generated by AI, which can sometimes be inaccurate or misleading, leading to incorrect decisions/learnings. Another significant issue is data privacy, as generative AI often requires and ingests large amounts of personal data, raising concerns about the security and privacy of student information and the serious consequences of improper handling or data breaches.
Furthermore, unequal access to AI technology can exacerbate educational disparities. Students and institutions with limited resources may not be able to fully utilise the advanced features of generative AI, thereby widening the gap between those who can afford such technologies and those who cannot, leading to significant inequalities in the quality of education received by students from different socioeconomic backgrounds (sciencedirect, Unveiling the shadows: Beyond the hype of AI in education).
In March 2023, more than 1,000 industry experts signed an open letter urging industry leaders to pause GenAI development so that the capabilities and dangers of systems such as GPT-4 can be assessed and mitigated if needs be. A New York attorney, who used ChatGPT to write a legal brief, had to provide an apology in court, as six of the cases cited in their submitted brief did not exist (Bridging the Gap Between Artificial Intelligence Implementation, Governance and Democracy: An Operational and Regulatory Perspective).
These disruptive examples underscore the need for careful implementation, regulation and guidance with respect to generative AI in the education sector to enhance the learning experience while minimising the potential risks. However, a recent ISACA (Information Systems Audit and Control Association) study on generative AI found that only 10% of organisations have formal, comprehensive policies in place for generative AI (Key Considerations for Developing Organizational Generative AI Policies).
Particularly in education, a field where research integrity is paramount, there must be precise regulation that guides generative AI and particularly with respect to the three cardinal principles: plagiarism, fabrication, and falsification. GenAI can replicate existing sources so accurately that sometimes it is difficult to distinguish from the original source (plagiarism). It can also generate fabricated data that can result in threats to research credibility (fabrication and falsification). Therefore, it is important for educational institutions to address these issues to maintain and improve high levels of research integrity.
Additionally, the increased uptake of AI will significantly affect the academic and career plans of students. In a new Inside Higher Ed-College Pulse’s Student Voice series survey, with more than 1,250 students responding from 49 four-year and two-year colleges, 14% said AI has influenced their academic plans a lot, and another 34% said somewhat. Also, regarding career plans, 11% said that AI influenced their plans a lot and 31% were somewhat influenced. More than two-thirds of students, 72%, believe that their colleges should be preparing them for working with AI. As for how institutions should be preparing them for AI and work, the top priority, according to 72% of surveyed students, is being taught the ethics of using AI. Therefore, colleges should reflect this request by updating curricula, offering interdisciplinary programmes, and establishing continuous learning pathways (Survey: How AI Is Impacting Students’ Career Choices).
According to OECD DIGITAL EDUCATION OUTLOOK 2023 Towards an Effective Digital Education Ecosystem, the status of regulation, guidance, and school responsibility, as of December 2023, is displayed in Table 1 below.
Table 1: Use of generative AI in education: guidance and practices, 2024
We can delineate AI’s impact on education on both a global and local level. Starting with the global level, it is largely recognised that generative AI is one of the transformative tools that will be useful in education. This is reflected in the set frameworks and guidelines drafted by organisations at a global level on its ethical and effective use:
- Guidance for generative AI in education and research (UNESCO) – In September 2023, UNESCO proposed guidelines concerning the ethical use of generative AI in education and research, which includes protection against violation of data privacy, age restriction setting, and mitigation policy frameworks. It also advises that teachers and students must be educated on how to use generative AI tools safely. It urges governments to quickly regulate the use of generative AI in schools so that it is integrated effectively into the education system.
- OECD DIGITAL EDUCATION OUTLOOK 2023 Towards an Effective Digital Education Ecosystem – In December 2023, the OECD explained how generative AI could be used within the digital education ecosystem. OECD Digital Education Outlook 2023 discussed how generative AI contributes as a push factor to innovation in education and sets out recommendations on how policymakers can grasp its benefits. However, to capture these benefits brought about by generative AI in an educational setting, comprehensive strategies are needed.
At the local level, various countries mentioned in Table1 have developed specific frameworks and guidelines to regulate and promote the use of generative AI in education.
- Australian Framework for Generative Artificial Intelligence (AI) in Schools – In November 2023, the Australian Framework for Generative AI in Schools highlighted the responsible and ethical use of generative AI tools for students, schools, and society. It provided guidelines for policymakers, school leaders, teachers, support staff, parents, and students on why they need to understand, use, and respond to generative AI in education. The framework aimed at improving educational outcomes with appropriate utilisation of AI tools by introducing safe and ethical practices, advancing equity and inclusion with 6 principles (see Figure 1) and 25 guiding statements.
Figure 1: Visualisation of Australian Framework for Generative AI in Schools
- Generative artificial intelligence (AI) in education – In October 2023, the UK’s Department for Education (DfE) published a policy paper guided by the government’s white paper on a pro-innovation approach to AI regulation. The paper stated that generative AI allows for a potential decrease in workload thereby allowing more time for teachers. At the same time, challenges arise from the content generated by generative AI. Since AI requires training based on input data, the students’ work is not to be used in the training of the model without their consent. Also, there must not be any malpractice with respect to these generative AI tools in any formal assessment. Students need to be educated on how to use technology appropriately and recognise its potential social impact. Furthermore, the DfE said it will provide support to primary and secondary schools in the delivery of a knowledge-rich computing curriculum.
These global/local guidelines and regulations are crucial for ensuring that generative AI is used safely and effectively in education. They aim to support teachers and students in leveraging innovative tools while maintaining ethical standards and protecting personal data.
The Irish context
Turning to the Irish context in relation to AI, recent studies conducted on behalf of the government suggest that around two-thirds of employment in Ireland will be affected by AI with up to one half of those jobs being at risk and the other half being enhanced. The Irish government welcomes these studies as it is calling for a national conversation on AI and the changes it will bring. Another study shows that multinational companies claim to use 30% more generative AI than indigenous, Irish organisations. The study surveyed 400 senior managers across both the public and private sectors in Ireland. The findings revealed an emergence of a “shadow” AI workplace culture, where staff are using publicly available AI tools regardless of any organisational acceptable usage policy. Such policies were only in operation in 2% of the firms surveyed despite 49% reporting some form of generative AI use already.
Call for responsible design
The introduction of generative AI has intensified concerns around misinformation and biased training data. It also presents new challenges in relation to authenticity, transparency, and intellectual property/data privacy. Getting to grips with these concerns and challenges is not only important and urgent but may also affect providers’ bottom line. A survey by The Economist Intelligence Unit found that 75% of executives oppose working with AI service providers whose products lack responsible design. Therefore, “FMOps” (Foundation Model Operations) / “LLMOps” (Large Language Model Operations) is an emerging area of research which includes bringing generative AI applications into production and monitoring them afterwards. Tasks include evaluating data and models, taking corrective actions, conducting audits and risk assessments, and establishing processes for continued model improvement. There will also be a need to expand the data infrastructure to govern and manage new, diverse, unstructured datasets to ensure alignment with privacy, security, and quality control as well as minimise the risk of sharing inaccurate or incorrect information that can lead to harmful or libelous outcomes. Such endeavours are critical if we want to ensure that the benefits of generative AI outweigh the risks.
Acceptable usage policy
Another dimension of generative AI regarding its use within a professional setting is establishing an acceptable usage policy for any organisation wanting to harness its capabilities in a responsible manner. Any such policy needs to address the concerns of different organisational stakeholders such as customers, regulators, and staff starting with why, i.e. motivate the business case for its use. It must also outline how any AI system adopted will be used safely, securely, and responsibly as well as how it will not be used. The policy should indicate how it chimes with existing regulation, e.g. data handling in conjunction with the General Data Protection Regulation within the EU, as well as existing policies within the organisation. Training for appropriate use and establishing a violation reporting mechanism is another important element of such a policy as is reinforcing the “human-in-the-loop” oversight aspect by identifying who will manage, audit, review, and update the policy and when. Moreover, it is important to provide a human point of contact in relation to this policy to address the concerns of any/all organisational stakeholders.
As developing generative AI solutions in-house can be extremely costly, most businesses and organisations are opting to adopt third-party generative AI tools or fine-tuning such models using their own data. This economic decision means that any business/organisation choosing to use a third-party generative AI tool must review the terms and conditions of use in advance and satisfy all business/organisational requirements through dialogue with any potential provider before greenlighting its use in a professional capacity. For example, ask a potential generative AI solution provider about any appropriate protections and assurances in place to address confidential information generated by your prompts. Also, ask any potential provider if they are willing to offer indemnities against IP/copyright/open-source (if commercially constraint) infringements and if these indemnities are subject to any contractual agreements. Furthermore, double-check the terms and conditions of these generative AI tools to understand who owns the IP (if any) in relation to the outputs generated.
Synthetic data
A final area of generative AI worth exploring in this paper is its use in relation to synthetic data generation. Its capability in this regard is threatening to revolutionise our approach to data creation and the training of AI models akin to how AI mastered the boardgames Chess and Go in recent decades. A key driver for synthetic data generation is due to a specific lack of readily available data to help us get better at specific tasks. If viable and done appropriately, synthetic data generation can help researchers overcome the biases and ethical dilemmas often inherent in real-world datasets as many domains suffer from a lack of quality data, especially when it comes to rare events or minority classes. LLMs can augment existing datasets, creating balanced and comprehensive datasets that improve the training and performance of machine learning models. That said, we should not assume the data generated is automatically accurate and representative given LLMs may be trained on biased data that is also lacking in both intra-class and inter-class diversity. These LLMs may also produce completely fictitious data that is totally disconnected from the desired model of representation/reality, i.e. the LLMs may “hallucinate”. Therefore, there is a pressing need to develop effective strategies to detect and address hallucinations in the context of synthetic data generation, ensuring the reliability and authenticity of the output. At the other end of the spectrum, it has been demonstrated that it is possible to extract specific information from the datasets used in training LLMs. Consequently, there exists a risk that synthetic data generation might inadvertently reveal elements of the underlying training data, which may raise some data privacy and ethical concerns thereby highlighting the need for careful consideration in the use and dissemination of synthetic data generated by LLMs.
Conclusion
Addressing the concerns associated with generative AI requires not just technological solutions, but also robust policy frameworks, ethical guidelines, and harmonised standards to ensure responsible use of data (synthetic or otherwise) and AI models.
[ii] https://langchain-ai.github.io/langgraph/
[iii] https://microsoft.github.io/autogen/
This paper is available to download here.