core: Fix handler removal in BaseCallbackManager (Fixes #30640) (#30659)

**Description:**  
Fixed a bug in `BaseCallbackManager.remove_handler()` that caused a
`ValueError` when removing a handler added via the constructor's
`handlers` parameter. The issue occurred because handlers passed to the
constructor were added only to the `handlers` list and not automatically
to `inheritable_handlers` unless explicitly specified. However,
`remove_handler()` attempted to remove the handler from both lists
unconditionally, triggering a `ValueError` when it wasn't in
`inheritable_handlers`.

The fix ensures the method checks for the handler’s presence in each
list before attempting removal, making it more robust while preserving
its original behavior.

**Issue:** Fixes #30640

**Dependencies:** None

---------

Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>

libs/core/langchain_core/callbacks/base.py

@@ -1007,8 +1007,10 @@ class BaseCallbackManager(CallbackManagerMixin):
         Args:
             handler (BaseCallbackHandler): The handler to remove.
         """
-        self.handlers.remove(handler)
-        self.inheritable_handlers.remove(handler)
+        if handler in self.handlers:
+            self.handlers.remove(handler)
+        if handler in self.inheritable_handlers:
+            self.inheritable_handlers.remove(handler)
 
     def set_handlers(
         self, handlers: list[BaseCallbackHandler], inherit: bool = True

libs/core/tests/unit_tests/callbacks/test_sync_callback_manager.py

@@ -0,0 +1,16 @@
+from langchain_core.callbacks.base import BaseCallbackHandler
+from langchain_core.callbacks.manager import BaseCallbackManager
+
+
+def test_remove_handler() -> None:
+    """Test removing handler does not raise an error on removal.
+
+    An handler can be inheritable or not. This test checks that
+    removing a handler does not raise an error if the handler
+    is not inheritable.
+    """
+    handler1 = BaseCallbackHandler()
+    handler2 = BaseCallbackHandler()
+    manager = BaseCallbackManager([handler1], inheritable_handlers=[handler2])
+    manager.remove_handler(handler1)
+    manager.remove_handler(handler2)