Here in below example the apex trigger is on “Account” object, to check duplicate Account Names.

Apex Trigger:

trigger AccountTrigger on Account (before Insert, before Update) {
     
    Map<String, Account> accMap = new Map<String, Account>();
     
    for (Account acc : System.Trigger.new) {
         
        //Make sure you don't treat account Name that isn't changing during an update as a duplicate.  
        if (System.Trigger.isInsert || (acc.Name != System.Trigger.oldMap.get(acc.Id).Name)) {
             
            //Make sure another new account isn't also a duplicate  
            if (accMap.containsKey(acc.Name)) {
                acc.Name.addError('An account already exist with same name.');
            } else {
                accMap.put(acc.Name, acc);
            }
        }
    }
     
    //Query to find all the Accounts in the database that have the same name as any of the Accounts being inserted or updated.  
    for (Account acc : [SELECT Name FROM Account
                        WHERE Name IN :accMap.KeySet()]) {
                            Account newAcc = accMap.get(acc.Name);
                            newAcc.Name.addError('An account already exist with same name.');
                        }
}

Here is the Test Class for the above apex trigger.
Test Class:

@isTest
private class TestAccountTriggers {

    static testMethod void AccountTriggerUnitTest() {
	
        Account acc1 = new Account();
        acc1.Name = 'ABC Corp Ltd';
        acc1.Type = 'Prospect';
        acc1.Industry = 'Banking';
        Insert acc1;
		
        Account acc2 = new Account();
        acc2.Name = 'ABC Corp Ltd';
        acc2.Type = 'Prospect';
        acc2.Industry = 'Apparel';
		
        try
        {
        	Insert acc2;
        }
        catch(System.DMLException e)
        {
        	System.assert(e.getMessage().contains('An account already exist with same name.'));
        }
    }
}